The Real Challenge of No Code for Data Pipelines

Upon launching the platform back in January we quickly found ourselves in talks with members of the no-code communities on Twitter and Reddit. Being a no/low-code platform this seemed natural and it felt like the right fit when conducting demos of the platform, discussing other products and common data challenges that are faced.

That was until we had a question from a call with No Code Devs (well worth checking out) where we were asked “What does it offer to no code users specifically?” We went through a few of the features again and discussed how we were different from existing integration-focused apps like Zapier or Parabola (both noteworthy products).

Since then, we’ve had other calls with no code whizzes and attended some webinars and other demos ourselves. It didn’t take long to realise that no-code is not a community of analysts or marketers who don’t code but instead builders who simply don’t want to. No code really is about building things, websites, services, applications that handle a specific integration between this and that, the list goes on… And what’s more, people are managing to do it all.

This last point is so fundamentally important and something that regular developers (such as myself), don’t (didn’t) understand, in that the very real solving of technological and business problems often (and proudly) without a single line of code is done by building. Simply put, no-code is about creation not about process.

So what does this mean for Data Pipelines?

We’ve realised that no code developers are not necessarily familiar with the concept of data pipelines, just as marketers are not. Instead, they’re used to terms like Data ‘Flows’ or ‘Connections’; the former being a series/step-based pipeline and the latter being often a simple one-way integration for example from Google Analytics to Google Sheets. This obviously presents a positioning challenge for us which will likely be ongoing for as long as our platform exists.

No code developers rarely have a use for big data (which our platform is built to handle) and often don’t store data in traditional databases, such as standard instances of MySQL, but instead, they use platforms such as Google sheets (or similar) where they can manually cleanse/append their data for use in their application or a specific tool that they’re trying to feed.

This means that our no code pipeline transformation features have someplace for automating the manual cleansing or appending of data which is often done by hand, even if the data is then just sent back to the original platform.

However, this also means that for many no-code users, adding to our pool of data warehouse or analytics systems connectors offers no further value.

What’s more, this means an unfamiliarity with SQL (and variant languages). This is a challenge for us in that our pipeline tools are defined around SQL functions and named accordingly. This makes us think, should we have a suite of tools (that can be toggled on and off) for no-code users, tools that are more descriptively named and designed to cleanse and organise data rather than analyse or manipulate?

Automation

If data then, for no-code users is used for building, feeding working applications instead of feeding models or reports, this calls into question whether our automation is in fact up to the task to service this.

Currently, we have two methods of triggering jobs. The first is a simple scheduler that allows users to schedule their pipelines to run from every minute to every 2 months and well beyond. The second method is via our API which will trigger a job to run when it receives a simple hit with the correct pipeline ID and account credentials.

The scheduler has proved itself to be easy to use but to increase adoption of the API among no-code users we need to make it as easy as possible with simple copyable JSON/cURL and/or make sure it is fully compatible with well known no code API tools.

Another possible feature would be data streaming which we’ve been thinking about for a while and would allow any updates to source datasets — in their native platforms — to run through the pipeline in real-time. This would be great for connecting backends of systems that would remain constantly up to date.

No-code is a unique challenge in itself, no other users have made us think about their challenges or our features more. We’re committed to staying in this space and making Data Pipelines the platform to shape and deliver their data so they can get their products and services off the ground.