Every data team knows this story: Some team member wrote some Python script that moves some data. You run it once and it works great! Now you want to run it on a schedule. Maybe every day? So someone sets up Airflow.

Then the real fun begins! You need to figure out how to monitor Airflow so you know if it’s still working. And you need some observability to understand why your scripts are slow. Also, you need to get secrets into the scripts in a secure way. And you need to figure out how to get code into production in an automated way. Oh ya, you also need a staging environment!

After much gnashing of teeth, you’ve managed to cobble together either home-grown or purchased services to solve all these problems. Congratulations! You’ve got a data platform! Kind of.

Airflow has quietly become the default backbone of the modern data stack. But that’s a problem.

Airflow is just a single piece of the overall puzzle. It’s a workflow orchestrator lacking all the key features you need to do real work.

It’s time for us to think hard about how and when we use Airflow.

Orchestration is a single platform feature

Let’s be clear: Airflow is good at what it was designed for. It orchestrates tasks. It lets you build DAGs and schedule them. That’s valuable, but that’s all it does.

• It doesn’t manage your environments.

• It doesn’t give you observability or secrets management 

• It doesn’t have error tracking.

• It doesn’t help you move from dev to prod safely.

• It doesn’t make your development experience better.

And because it doesn’t do those things, teams end up having to build or buy all these features and services. In my work at Tower, I hear this story over and over again.

For instance, one of Tower’s design partners set up Airflow to do some basic AI training. It worked great until at some point six months later they figured out…it hadn’t actually been running for weeks! The ensuing firedrill made them reevaluate their whole strategy around Airflow. What they needed was a platform with complete monitoring and alerting capabilities.

You end up rebuilding everything around Airflow

Teams need these features and they often need them in a crisis. For instance, most people onboard monitoring for their Airflow only after a major outage–typically one they realize hours or days after Airflow stopped working.

It’s on you to run around and sort all this out. There are vendors you can integrate together to help, but the most important thing is you have to allocate your (very) precious development time to work on a non-differentiating platform. One that looks exactly like one you’ve built before, I’m sure.

And after you’re done, despite your best efforts, you’re now running a fragile, bespoke data platform that nobody wants to maintain.

Developer experience is an afterthought

Airflow is understandably really focused on running tasks (your code) across its cluster on a schedule. How does that code get there? Well, that’s up to you my friend. If you’re self hosting Airflow, then simply put it on the filesystem…somehow. There exist plugins for you to integrate your GitHub with Airflow, but it’s again something you have to find and manage yourself.

But shipping code is only a small part of the overall development process. Take the development environment, for example: At the highest level, development environments should look a lot like production to avoid hidden environment differences that will bite you when your code lands.

There’s no recommended pattern or prescribed solution for managing Airflow in development. Maybe you can mock Airflow in your dev environment, but ideally your platform is more deeply integrated with your development environment and workflow.

Data teams need real platforms that speak Python

If you’re building with the Pythonic composable data stack, you need more than just a scheduler to get your code out the door. You need a complete platform.

• Natively understands Python, not just as a plugin.

• Manages environments, secrets, and dependencies automatically.

• Gives you structured logs, runtime metadata, and monitoring out of the box.

• Supports both batch and interactive services.

• Lets you test, ship, and observe your pipelines with the same confidence as any backend service.

The future of data engineering looks more like app development: fast, iterative, testable, and ergonomic. Airflow alone clearly doesn’t get you there.

Stop falling into the Airflow platform trap

Airflow is not the enemy. It’s a great tool. But it’s not your platform and it was never meant to be.

The more time your team spends gluing together infra around Airflow, the less time you're spending solving real data problems. It’s a distraction, a tax, and a source of hidden complexity.

It’s time to stop pretending Airflow is the foundation you need. It’s time to look for something built for this decade—not the last one.

We understand this problem

At Tower, we’re building the platform we wish we had—serverless Python infrastructure purpose-built for data engineering. It’s everything Airflow isn’t: ergonomic, fast, native to Python, and production-ready out of the box.

Deploy your dbt, dltHub, or custom Python code without reinventing the platform. Because you shouldn’t have to duct-tape your stack just to get things done. Sign up for the Tower private beta today to give it a spin yourself!