Airflow vs. Prefect

Late to the party, but I wanted to mention that the way Prefect decoupled the DAG / scheduling / context is much better for my multi-tenant saas product. Airflow did not seem to be a good match for my use case, where I need to have a set of tasks per-tenant that can be scheduled by users with the proper permissions (per tenant), and then a set of different tasks at the user level that can also be scheduled (per user).

Airflow just did not fit that model at all. Prefect supports that just fine.

There are fewer steps to setting up Airflow locally compared to Prefect. There seems to be encouragement with Prefect to use remote servers instead of running things locally.

Airflow has been around since late 2014 and has ~800 developers contribute to it. It's hard to search for an issue with Airflow without coming across something helpful. That sort of community support will take time to develop with Prefect. Prefect's server repo will be celebrating its first birthday on Monday.

Prefect looks to be a project of a few developers aiming to make a hosting business out of the offering. I'm not certain if their ecosystem of developers and experts will grow to the size of Airflow's in the near future.

Prefect has taken advantage of recent language additions in Python 3 and its source is a joy to read through. They use Black to ensure the code formatting is consistent. They've decoupled the server, web UI and their answer to DAGs into 3 code bases.

Prefect seems aimed at those that want a managed solution to their data pipelines. The need for skilled engineers for data engineering has exploded in recent years and a lot of businesses struggle to find the senior staff they need to do things right the first time. I suspect Prefect will find some success removing certain steps and complexity in building data pipeline from less senior engineers which should help more businesses become data-oriented.

Very few Airflow competitors did a good job at decoupling from any one task runner, metadata backend, etc.. and brought few features to the table that Airflow lacked. Prefect's UI looks fantastic, the code base is really tidy and if they could ever turn the install process into a simple 'pip install prefect' with no need for Docker, K8s, npm, font awesome tokens or any external communication I'd be very excited to dig deeper into the project.