Every data engineer has war stories. Pipelines that silently failed for weeks. “Small” Spark jobs take forever. Tools that were easy to get started with but made you lose sleep (and money) over time. Most of these stories share a common theme: Decisions were made early that didn’t seem like a big deal… until they were.

These are what I call traps and they can be absolute hell for data engineering teams. They’re things that you develop an intuition for–more art than science–that the gray beards try to teach the young folks. In this post, we’ll take a look at the four most common traps I see data engineers walk into (especially at growing companies) and why they’ll end up hurting you later.

1. You assume you have big data.  

Trap: Designing for scale you don’t have locks you in to an expensive and complex architecture you don’t need

Let’s be honest: most companies don’t have big data. They have medium CSVs at worst, and small Postgres on average.

But how many times have you gone to a meetup to see architectural diagrams built around Spark, Kafka, or Kubernetes infrastructure in the first six months of a company’s lifecycle? This becomes a complexity tax for the team. It’s a huge amount of operational overhead with convoluted deployments and brittle pipelines. All for data volumes that would fit in a Pandas dataframe.

The future is all about small data.

Why it bites you:

Prematurely optimizing for scale means you’re building for imaginary constraints. You’ll burn engineering time maintaining infra that’s irrelevant, while delaying the stuff that actually moves the business forward: reliable pipelines, clear observability, fast iteration.

What to do instead:

Use boring tech until it hurts. Start with single-node Python. Use DuckDB. Use batch jobs. Add complexity when your data—and your team—demand it, not before.

2. You make your orchestrator your platform.  

Trap: Ignoring this fact, you’re left to backfill all the other features a real platform is meant to give you.

A lot of teams adopt Airflow, Prefect, or Dagster and feel like they’re 80% done with their data platform. See that? There’s the trap. Orchestration is just a scheduler with dependencies. It doesn’t solve packaging, environments, secrets, CI/CD, observability, deployment, or loads of other problems.

Why it bites you:

You’ll spend months duct-taping together an actual platform around your orchestrator. I guarantee you, you’ll write custom Kubernetes scripts, debug dependency hell, and manually build tooling that already exists in modern developer workflows.

What to do instead:

Treat orchestration as just one feature of your platform—not the platform. Look for systems that handle packaging, execution, and monitoring out of the box. Or if you build your own, plan to invest in real platform engineering.

3. You ignore the development process.  

Trap: Not developing a structured process for getting code from laptops to production doesn’t scale beyond one person.

Most teams have two environments they’re concerned with in the early days: Their local `pip install` environment and production. Every deployment becomes a leap of faith when moving code between them when you don’t have, at the very least, CI and a staging environment. And you do have tests you run, right? Right?!

Why it bites you:

Without structured development workflows, you can’t safely test changes, onboard new team members, or manage scale. Every change is high-risk, debugging is guesswork, and reproducibility is a dream.

What to do instead:

Adopt software engineering practices. Use version-controlled code. Make dev/prod parity the norm. Add local emulation, isolated environments, and observability. Think like a software team—not just a data team.

4. You confuse the build vs buy equation.

Trap: Building things is actually just buying things with extra steps.

Some teams build everything themselves—to avoid vendor lock-in, to stay flexible, or just out of habit. Others go all-in on a monolithic platform and hope it “just works.” Neither approach works well, but the former is where the trap is unless you manage the roadmap very carefully. No one wants to bleed cash in the form of salaries to build and maintain table-stakes, non-differentiating platform features.

Why it bites you:

Building everything yourself leads to reinvention, slow progress, and high maintenance. Buying the wrong monolith traps you in an opinionated stack that won’t grow with your needs—or costs a fortune to escape.

What to do instead: 

Buy commodity infrastructure. Build differentiated logic. Use open standards where you can (like Iceberg or dbt), but don’t be afraid of opinionated tooling—if it lets your team move faster. The goal isn’t flexibility—it’s leverage.

What experience teaches

Data engineering is full of seductive shortcuts and familiar defaults. But default choices often carry hidden complexity—and those are the things that come back to bite.

At Tower, we’ve built a platform that avoids all four of these traps. It’s designed for Python-first data teams who want production-grade infrastructure without the yak shaving. Serverless, environment-aware, orchestration-ready—but never over-engineered.

Want to avoid these traps? You don’t have to do it alone. Let’s talk.