We spent a week in San Francisco at the Small Data SF event. As Joe Reis quoted William Gibson, “the future is here, just not very equally distributed.” That sums up our experience.
It is not just what San Francisco itself represents: driverless cars and vibe-coding apps are normal here. It is also what we saw at the event. The talks covered practical applications of small data, small teams, and small queries to produce the best results.
We noticed many trends at the Small Data SF conference; here are some worth covering.
It's not about making things bigger
Most of the blog posts and showcases you’ll see at events and online will be about moving, transforming, and managing big data. It makes sense: moving a mountain is much more impressive than moving a pebble.
Yet, maybe because of the bias toward “small data” or because of how tech works in general, making things smaller and more manageable is becoming a greater focus in the industry. Ranging from decisioning platforms to Small Language Models, many companies are trying to figure out how to make things smaller. Even the CEO of 23andMe, which manages genetic data, was discussing how they reduce their massive dataset to make it more accessible.
Tech often starts big and then gets smaller. Many major advances in both consumer and industrial tech come from miniaturization. Computers used to fill a room, but now they fit in your pocket. The trend is toward making things smaller and more efficient. The future of data may be about handling big challenges with small solutions.
AI leading the big (and small) discussions
AI is on everyone’s mind these days, and the Small Data SF event was no exception. People were wondering how it would change the data engineering industry. As Joe Reis said in his keynote, “We know everything is changing, but how?”
Data engineering is different now than it was a year ago. Agentic AI, LLMs, and having them code for you have changed the nature of the job. Engineers use Claude to build pipelines; that’s why we built our own MCP server. Small teams are solving bigger problems for their organizations than ever.
The talk on the floor was no different. We heard from an Enterprise Data Architect at an 150-year old construction company how he is managing his significant IoT measurements dataset by operating a half-dozen agents in Claude Code, defining agent roles for Data Analysts, Product Managers, and Data Engineers, and letting Claude automate database management. His company decided to save on IT by reducing his team to him and a Data Analyst. He decided to stay sane by “hiring” 5 Claude agents. In conversations like this one starts feeling that the future is about enabling a small team to do big things.
At Tower, we build tools that help even the smallest teams get things done. The feedback we received at the event was encouraging. We’re right on time to help people with limited resources accomplish big goals.
What does the future look like?
A lot of folks at the conference were discussing how we can enable users to use natural language prompts to create applications. How do we get our tools and non-coding friends to achieve something as simply as saying, “I want to create a pipeline?”
At Tower, we’re working on a few ideas, like enabling agents to use our systems via the MCP protocol.
Final thoughts
Small Data SF was an opportunity to talk to people about what we’re building.
We’re heading back to Berlin now, starting to prepare for the upcoming PyData Warsaw and Boston events (more on this shortly).
Follow us on social media, sign up for our newsletter, and look out for us at events in Europe and the US.
Until then, we’ll keep building for the small teams doing big things.