How to update a scheduled pipeline

To update a pipeline's transformations (steps)

It is possible to make changes to a pipeline once it has been scheduled and running.

To do this, go to the pipeline builder and make your changes. That's it. There is no need to reschedule the pipeline, your changes are live and will affect the output of the next pipeline run.

If you want to test your changes immediately, go to the pipeline's Schedules view and click the Run Now 'bolt' icon. If you would like to test them without running the main schedule, feel free to add a new schedule that uses a different output destination. There is no limit on the number of schedules you can have per pipeline.

To update a pipeline's schedule(s)

Schedules cannot be updated but they can easily be deleted and re-created with different parameters.

To update data sources or destinations used by existing pipelines

Data connections are identified by name. Names are unique within each connector type. So e.g. you cannot have two SQL database connections with the same name. This makes it easy to switch between instances of databases without having to update each pipeline individually.

You can update or delete data connections at any time. If they are in use by any of the currently existing pipelines, you will get a warning. Note, that as data connections are defined globally, changes to a data connection will affect all pipelines depending on it.

If you want to change the data source parameters for a single pipeline only, do the following:

• create new data connection
• find the pipeline in the main dashboard view and click Duplicate
• in the resulting JSON definition find the source dataset you want to change and update the name / type to the one you have just created
• create a new pipeline from definition
• schedule the new pipeline
• delete (or keep and unschedule) old pipeline

Note: duplicate feature is only available for Processor tier

To change destination connection for a single pipeline:

• create new data connection
• unschedule pipeline
• schedule pipeline using the newly created destination connection