Building a big data platform: Lesson 1

The previous 9 months have been unprecedented, challenging and for the most part completely forgettable. However, we’ve been uniquely fortunate to have had a project that’s kept us busy outside of our day jobs, up at night, through eventless weekends, and just about the dullest Christmas in memory.

It’s called Data Pipelines. The elevator pitch is “a no-code, big data platform with built-in SQL tools and connectors for AWS, Google Cloud, and more.”

We spent just over 6 months building it and we launched it a month ago on datapipelines.com, here’s what we’ve learnt.

Lesson 1. Focus on the customer, they alone will inform the truest values of your product.

This one snuck up on us.

We developed an MVP (minimum viable product) within three months and by the fourth had tested it with an agency that used it for some clients, generating reports from Amazon S3 into Google Sheets. The connectors between the two data platforms worked flawlessly; the SQL tools did all the transformations they were currently coding and within a week they had several large ETL jobs fully automated. They were happy, we were very happy.

It was at this point, four months in, that we should have put all our energy into getting more users, but we didn’t know it yet. Instead, we were inspired by the use of the platform (exactly how we had designed it) so we continued to develop. Over 4 weeks, we pushed to improve the UI, tweak the SQL tools, and add some accessibility features which we felt the platform desperately needed.

During this time we’d also managed to acquire another agency user who required a Google BigQuery connector, something that was on our ‘Future Connectors list’ anyway so we built it in for them later that week.

By the weekend, the new agency was our second-biggest user.

Damn.

Finally, it dawned on us that the new connector was the only value we had added in the last month and most strikingly we hadn’t had to deliberate it into existence; it had been a no-brainer, driven fully by the customer.

We decided to finish off our current round of additions, knowing that we’d experienced a common trap in product development and grateful that we’d experienced it early on. The extra additions we made in these early stages, we still regard as important, only costing us time. That said, our early December soft launch reminded us that finding time to fiddle with CSS is easy but a warm lead two weeks before Christmas is far more elusive.

Our lesson was an important one. Moving forwards we will not to put growth on hold to add features that we assume will add value, rather focus on growing the userbase and let them drive the demand for changes to the product.

We hope you’ve enjoyed Lesson 1. Stayed tuned for more as we turn our heads to our growing user base.