Document all active DAGs, task count per DAG, dependencies, and operational characteristics (execution frequency, typical runtime, failure rates). Identify which DAGs are critical vs. experimental. Note any custom operators, plugins, or hooks you've developed. Export DAG metadata from Airflow's database or UI.
⚠️ Watch Out For:
- Dynamic DAG generation may not be immediately obvious—search for DAG creation loops
- Complex branching and conditional logic can be harder to track than linear task sequences
Study Prefect flows, tasks, and blocks. Understand how Prefect's state model differs from Airflow's (Pending → Running → Success vs. Airflow's queued/scheduled/running/failed). Review Prefect's async/concurrent execution model. Get familiar with Prefect's UI and API.
⚠️ Watch Out For:
- Prefect's flow paradigm is different enough that direct DAG-to-flow mapping often misses optimizations
- Async execution in Prefect can expose concurrency bugs hidden in Airflow—test thoroughly
Install Prefect. Create a Prefect workspace. Decide on deployment strategy: Prefect Cloud (managed, recommended) or self-hosted Prefect Server. Configure authentication and API keys. Set up agent/worker infrastructure for task execution.
⚠️ Watch Out For:
- Prefect Cloud setup is simpler; self-hosted requires Docker/Kubernetes knowledge
- Agent/worker pools must be configured for different execution environments (cloud, on-premises, Kubernetes)
Select the simplest Airflow DAG (ideally a linear sequence of 3-5 tasks). Rewrite it as a Prefect flow using @flow and @task decorators. Implement the same logic but structured for Prefect's model. Test locally. Deploy to Prefect.
⚠️ Watch Out For:
- Airflow's XCom feels magical compared to Prefect's explicit task returns—getting used to the verbosity takes time
- Prefect's error handling is simpler but requires rethinking exception management
Create deployment definitions for each flow. Configure schedules (cron, interval, or event-based). Set up task caching and state persistence if needed. Configure retry logic for transient failures. Test schedule triggers in Prefect UI.
⚠️ Watch Out For:
- Prefect's schedule semantics differ from Airflow's—carefully map your existing schedules
- State persistence across runs requires proper task result caching configuration
Set up Prefect notifications (Slack, email, webhooks) for flow run failures. Configure work pool sizing for concurrency. Set up flow run filters for alerting on critical runs. Monitor execution logs in Prefect UI. Document alerting strategy.
⚠️ Watch Out For:
- Prefect's notification system is simpler than Airflow's but less granular—plan for custom logic if needed
- Work pool sizing affects cost and concurrency—tune based on load testing
Progressively rewrite remaining Airflow DAGs as Prefect flows. Start with simple ones, work toward complex. For each DAG, validate that outputs match Airflow. Document any patterns that don't translate well. Accumulate confidence through iteration.
⚠️ Watch Out For:
- Complex branching and dynamic task generation require different patterns in Prefect—plan ahead
- Some Airflow operators may not have direct Prefect equivalents—research alternatives early
Deploy Prefect infrastructure (agents/workers) to production. Configure environment variables, secrets, and resource constraints. Set up authentication for data source connections. Test end-to-end flow execution in production. Monitor for any issues.
⚠️ Watch Out For:
- Agent/worker connectivity to data sources must be validated—network firewall rules, VPN access, etc.
- Secrets management in Prefect differs from Airflow—ensure sensitive data is properly masked
Keep both Airflow and Prefect running in parallel for 1-2 full execution cycles. Compare outputs: task run times, success/failure rates, data accuracy. Validate that Prefect flows produce identical results to Airflow DAGs. Fix any discrepancies before full cutover.
⚠️ Watch Out For:
- Timing differences between Airflow and Prefect can complicate output comparison—align schedules temporarily
- Performance characteristics may differ (Prefect async execution may be faster)—account for this in validation
Once Prefect passes validation, disable Airflow scheduler. Keep Airflow running read-only for 2 weeks for reference. Update team documentation and runbooks. Remove Airflow from production infrastructure. Archive DAGs for historical reference.
⚠️ Watch Out For:
- Partial Airflow job runs may still be in progress during cutover—verify all are complete before stopping scheduler
- Team ramp-up on Prefect continues after cutover—expect questions and edge cases to emerge
