Audit every DAG in your production Airflow instance. Document task count, dependencies, frequency, and integration points. Identify which operators you rely on (BashOperator, PythonOperator, SQL operators, etc.). Note any dynamic task generation, branching logic, or conditional execution. This inventory determines migration scope and complexity.
⚠️ Watch Out For:
- Hidden DAGs in subdirectories may not be immediately obvious in the DAGs folder
- Complex branching and XCom dependencies are harder to trace than simple linear DAGs
- Airflow plugins and custom operators may not have Dagster equivalents
Rather than task-by-task mapping, redesign your pipelines around data assets. Identify source systems, transformations, and outputs as distinct assets. Define asset dependencies (asset A produces input for asset B). Sketch the asset lineage graph. This conceptual shift from tasks to assets is the core of the migration.
⚠️ Watch Out For:
- Airflow's task-centric mindset doesn't map 1:1 to Dagster's asset-oriented model—rethink the architecture
- Circular dependencies in Airflow are errors; in Dagster, they're also errors but caught earlier
Install Dagster and create a project structure (poetry init, pyproject.toml with dependencies). Set up a local Dagster instance with SQLite backend for testing. Configure Dagster's code location to point to your assets. Ensure the Dagster UI runs locally without errors.
⚠️ Watch Out For:
- Dagster's dependency management differs from Airflow—manage versions carefully
- Code location reloading in Dagster can be slow compared to Airflow's dynamic DAG parsing
Select a simple, non-critical DAG to be your pilot. Rewrite it as Dagster assets and ops. Define inputs/outputs explicitly. Use Dagster's @op and @asset decorators. Implement the same business logic but structured around asset production. Test locally with the Dagster UI.
⚠️ Watch Out For:
- Airflow's XCom pulls feel magical; Dagster requires explicit input/output definitions—more verbose but clearer
- Error handling differs: Airflow uses try/except in tasks; Dagster uses Failure/DynamicOutput types
For scheduled assets, define job definitions with Dagster's job decorator and schedule assets with @daily_schedule or @sensor decorators. If your Airflow DAG uses sensors (e.g., S3KeySensor), rewrite them as Dagster sensors. Test the schedule in local development using Dagster's schedule run UI.
⚠️ Watch Out For:
- Dagster's @sensor requires more explicit code than Airflow's sensor operators
- Timezone handling differs—ensure schedules align with your expected run times
Define Dagster asset checks (tests that verify properties of your assets). Write Python functions that validate data (row count checks, NULL counts, freshness checks). Integrate these as part of asset definitions. This replaces ad-hoc Great Expectations tests in Airflow with first-class Dagster checks.
⚠️ Watch Out For:
- Airflow's Great Expectations integration was loose; Dagster's asset checks are tightly integrated but require learning the API
- Deciding which checks to automate vs. which to alert on takes iteration
Rewrite remaining DAGs one by one, starting with simple ones. Test each migration job locally with Dagster's UI. Validate outputs match expectations. Document any Airflow patterns that don't translate (e.g., multiple-outputs branching). Accumulate confidence as you go.
⚠️ Watch Out For:
- Don't try to migrate all DAGs at once—prioritize by criticality and complexity
- Dynamic task generation in Airflow (e.g., DAG from list) requires different patterns in Dagster (DynamicOutput)
Deploy Dagster to your production environment (Kubernetes, cloud VM, or managed service like Dagster Cloud). Set up the same data warehouse connections, secrets management, and resource configurations as Airflow. Configure a production code location that points to your migrated jobs. Set up monitoring and alerting.
⚠️ Watch Out For:
- Secrets management in Dagster differs from Airflow—ensure sensitive variables are properly configured
- Executor selection (in-process vs. Kubernetes) affects how jobs run—test thoroughly before production
Keep Airflow running in parallel with Dagster for 1-2 full run cycles. Compare outputs from both orchestrators for the same source data. Verify that Dagster's runs complete at expected times and produce identical results. Use Dagster's event monitoring to catch failures early.
⚠️ Watch Out For:
- Timing differences between Airflow and Dagster can cause test data misalignment—ensure data freshness matches
- Partial failures in Dagster expose data lineage gaps not visible in Airflow—fix these before full cutover
Once Dagster passes full validation, remove Airflow from the critical path. Keep Airflow running read-only for 2 more weeks for reference. Notify the team of the change and update documentation. Remove Airflow's scheduler from your infrastructure. Archive Airflow's DAGs for historical reference.
⚠️ Watch Out For:
- Team members may still reference Airflow documentation—update wikis and runbooks early
- Partial job runs in Airflow may still be in progress during cutover—verify all are complete before stopping scheduler
