If you are one of those rare individuals who can read or understand spoken word, and you happen to work in the software engineering profession, you've most likely come across the news that major tech companies have stopped hiring developers and that major-league CEOs believe that AI will replace engineers starting right about this year.

Since I am a CEO as well, albeit of a small tech startup, I got really excited by the idea of being able to do what startup CEOs usually do (expensive dinners with investors) and build the product all by myself. It's tough finding good engineers; they tend to have opinions that do not always align with mine, and so on. Wouldn't it be nice if I could pay $20 monthly for a  Claude subscription, command it: "Claude, build Tower!" and watch the construction from my winter home in Madeira. 

I am an action-driven guy, so I've decided to act on this great idea and picked a task I was confident "the AI" could quickly solve. 

You see, a while back, I asked my co-founder Brad to write a Tower example using the Polars library and an Iceberg table. I pointed him to a dataset of Japanese trade data and asked him to write an expression that transforms detailed stats about traded goods into an aggregated report of monthly exports.

Brad probably thought that my ask was silly. He was working on hard platform stuff in Tower, so he thought about it for a minute and wrote this Polars code:

icetable = catalog.load_table(iceberg_table_name)


# create a Polars dataframe
df = pl.scan_iceberg(icetable)


df2 = (
   df.filter(
       (pl.col("exp_imp") == '1')
   ).collect().select([
       pl.col("ym"),
       pl.col("Value")
   ]).group_by(['ym']).sum().sort("ym")
)

This expression creates a Polars dataframe that sums up Japanese exports by calendar month. 

When you print the dataframe, you get a table like this:

 ┌────────┬──────────────┐
 │ ym     ┆ Value        │
 │ ---    ┆ ---          │
 │ i64    ┆ i64          │
 ╞════════╪══════════════╡
 │ 201701 ┆ 10841884462  │
 │ 201702 ┆ 12694245118  │
 │ 201703 ┆ 14455996554  │
 │ 201704 ┆ 12660379804  │
 │ 201705 ┆ 11703232472  │
 │ …      ┆ …            │
 │ 202007 ┆ 91276046740  │
 │ 202008 ┆ 88962776653  │
 │ 202009 ┆ 102920398105 │
 │ 202010 ┆ 111618743769 │
 │ 202011 ┆ 91704935670  │
 └────────┴──────────────┘

I was happy with this example initially, but then Tower users started asking questions. What did the column labels "ym" and "Value" mean? And could we not make the report MECE (mutually exclusive and collectively exhaustive) and add the calculation of imports as well? 

I went to Brad and found him deep in coding the next version of Tower. 

"Brad, could you …"

Brad's sad blue eyes said everything. This time, Brad could not. 

Feeling frustrated that I could not finish two straightforward things quickly - adding the imports column and changing labels - I commiserated that this was not SQL. If it were SQL, I would know what to do, but with this Polars library, it was unclear how to sum imports separately from exports. I was still learning Polars.

Two beautiful things happened to me then.

This story has a happy ending. If you want to take a shortcut, try the Tower beta and talk to one of our co-founders about your use case, please contact us. 

First, I remembered a talk (recording, post)  I gave with Simon Rosenberger, the Head of Data Platform at Taktile, a leading decision intelligence platform for FinTechs.  

Simon had a similar problem but faced it from the other side. He had a small data team serving the needs of business and platform engineering users at Taktile. Simon constantly got requests to add a field here and change a label there, and he could not take care of all of them. So, instead of asking for more headcount, he looked for tooling that made his existing data team 10x more impactful. He envisioned a future where everyone at Taktile, with just a little Python knowledge, could make small but very relevant changes in the pipelines. 

Simon's users and I were in the same boat—we wanted someone like Simon or Brad to create the initial version of a data app and then have the tooling to make small incremental changes to it without taking too much of our engineers' time and driving them mad in the process. 

Simon developed a workable solution (demo) that included Tower as the runner for Python apps and dltHub as their base framework. As sweet and very meta—use Tower to modify Tower apps—as it would have been to take Simon's solution as is, I had to evolve it because I knew what output I wanted to have but did not know how to change the app's code. 

Then, the second beautiful thing happened.  

I came across a post from our partner dltHub about using Cursor - a new, buzzy IDE - and its AI assistant to speed up pipeline code development. In the post, Marcel from Mooncoon gave excellent tips on configuring Cursor for maximum effectiveness.

Having some time, I tried it out on my task. And Cursor was indeed great. 

It often felt like Cursor was reading my mind and anticipating my next action. A big part of its magic was its use of the Claude 3.5 Sonnet LLM from Anthropic, which is much better at developer-y tasks than good old ChatGPTee.

I loaded my Python app in Cursor and instinctively selected the dataframe expression I wanted to modify. 

In the pop-up chat window, I typed my prompt: "Change this Polars expression to output a table with the ym month, the sum of all exports, and the sum of all imports."

After some grokking, Cursor proposed the following code:

df2 = (
   df.select([
       pl.col("ym"),
       pl.col("exp_imp"),
       pl.col("Value")
   ])
   .group_by(["ym", "exp_imp"])
   .agg(pl.col("Value").sum().alias("total_value"))
   .pivot(
       values="total_value",
       index="ym",
       columns="exp_imp",
       aggregate_function="first"
   )
   .rename({"1": "exports", "2": "imports"})
   .sort("ym")
)

It even summarized the changes for me:

Key changes made:

Modified the select to include exp_imp column
Used pivot to create separate columns for exports (exp_imp='1') and imports (exp_imp='2'

A great thing about the process was that Cursor used my entire data app to generate the proposed changes, including the comments before and after the selected expression.

# see this dataset for code values
# https://www.kaggle.com/datasets/zanjibar/japantradestat
# exp_imp: 1 is export, 2 is import
# hs2 to hs9 are goods classification codes
# Q1, Q2 is quantity. http://www.customs.go.jp/toukei/sankou/howto/yougo_e.htm
# Value is in 1000s of Yen

That's how it recognized that "1" stands for exports and "2" for imports.

As is customary for software development flows, Cursor showed me the changes in my code and allowed me to accept them individually, giving me a sense of control.

Excited about the future and where humanity was going, I ran the app on Tower.

The future was not as bright as it initially seemed because of a runtime error.

Exception has occurred: AttributeError
'LazyFrame' object has no attribute 'pivot'
 File "/Users/datancoffee/…/tower-examples/06-iceberg-analyze/iceberg_analyze_with_polars.py", line 42, in <module>

Here is the offending line.

At this point, I needed to do things old-school and google the error message ""'LazyFrame' object has no attribute 'pivot'."

I got some help on the not-quite-dead-yet StackOverflow and still-alive-and-kicking GitHub.

It turns out that without calling collect() on a Polars dataframe, effectively loading it into memory, you can't do pivot() on it because:

"Pivot will not be exposed in lazy because we cannot determine the schema without collecting the data first. So you must do the collect explicit."

So, I explicitly added the collect() call before the pivot().

df2 = (
   df.select([
 … bunch of code …
   .collect()
   .pivot(
… more code …
)

This indeed did the trick. The app ran and produced the desired breakdown of imports and exports, even printing it in a neat table.

DeprecationWarning: The argument `columns` for `DataFrame.pivot` is deprecated. It has been renamed to `on`.
  .pivot(

=== Japan Trade Statistics by Month ===
Month    | Exports (1000s Yen) | Imports (1000s Yen)
------------------------------------------------------------
201701  | 13,051,910,392 | 10,841,884,462
201702  | 11,085,193,530 | 12,694,245,118
201703  | 13,249,050,260 | 14,455,996,554
…
202009  | 64,444,745,592 | 72,649,692,780
202010  | 68,354,786,076 | 78,789,701,484
202011  | 57,475,473,300 | 61,136,623,780

How did I get the nice formatting? It was another neat suggestion by Cursor's LLM. It proposed adding this formatting code instead of plain old print(df2)

print("\n=== Japan Trade Statistics by Month ===")
print("Month    | Exports (1000s Yen) | Imports (1000s Yen)")
print("-" * 60)
for row in df2.iter_rows():
   month, exports, imports = row
   print(f"{month}  | {exports:,} | {imports:,}")
print("=" * 60)

If you paid attention, you probably saw the yellow flags in the generated code because it used an already deprecated method .pivot. Instead, .on() should have been used.

DeprecationWarning: The argument `columns` for `DataFrame.pivot` is deprecated. It has been renamed to `on`.
  .pivot(

This is an artifact of the LLM model training delay—the LLM was trained months or possibly years ago, and since then, the Polars library has deprecated the method.  

While not critical today, my little app would fail with a runtime error in several months if I chose to update Polars to the then-current version.

Conclusions

Data engineers are still safe in their jobs. For users who roughly know what they want to have in code, personal AI coding assistants are a great productivity hack. It's not a 10x revolutionary improvement, but more like a 30% boost because you still need to ask smart prompts and be able to debug potential errors in generated code.  

So, while AI assistants won't replace data engineers just yet and won't make them personally 10 times smarter and faster, they do create a 10x opportunity elsewhere. AI assistants can help turn a casual hobby coder into someone who can improve a company's data and code assets. They can upgrade someone with SQL skills into someone who can make changes to Polars dataframes. 

Companies will still need data teams, but they can be small, like Simon's team at Taktile or Marcel's team at Mooncoon. After the data team creates the initial code artifact, all their data analyst and non-data engineer internal customers can update pipelines in a self-serve fashion, with the data lead serving as a PR reviewer and the final arbiter of the company's code base. 

There are other lessons to be learned here, too. Claude-generated code led to 1 fatal error and 1 warning (using a deprecated API, so a future fatal error). The first fatal error was a runtime error and could not be caught before execution because it was triggered by the state of the object at run time (the Polars dataframe pointed to an Iceberg table, and the state of the object was "lazy").

Runtime errors are just one aspect that needs to be collected during execution. Was the generated code performant? Did it respect the data permission boundaries or try to ignore them?

It's clear that collecting logs and metrics while running LLM-generated code is as important as generating the code itself. These logs and metrics need to be analyzed, insights gleaned, and fed back into the LLM code generator so that the apps can evolve.

This is what Tower is about. Regardless of who writes the source—a human, an LLM, or a combination thereof—we strive to help data teams and their customers improve their development and runtime experiences. Contact us to talk to a Tower founder about trying our beta.

Exciting times ahead!