First in a series on VectorFlow. Where the idea came from, what I learned, and what I got wrong.

I came back from vacation to the news that dbt was merging with Fivetran.

I should have seen it coming. I didn’t. And it changed what I was working on.

The stack that made data teams fast

If you’ve worked around data movement, you know that hiring a team to hand-build pipelines stopped being the default years ago. Moving Postgres into a warehouse. Pulling an ERP system into MySQL. Whatever the source and destination, that used to mean a dedicated team and months of work before anyone saw a single data point in a visualization tool.

The modern data stack ended that. It had four pillars. Integration moved data from source to destination. Transformation turned moved data into something usable, which in practice meant dbt, with no viable competition. Orchestration scheduled pipelines once they had dependencies, which in practice meant Airflow. Visualization was where the data finally got looked at.

Integration was the most contested pillar, and it’s the one Airbyte competed in. Fivetran led it as both the most popular option and the most premium one. Airbyte came second as the value option, which made it a legitimate threat to Fivetran’s ability to scale toward an eventual IPO.

With those four pillars, a small data team could stand up working pipelines in a month. The same work used to take a year.

Consolidation was always coming

Every crowded market ends this way. Food delivery had Grubhub, Uber Eats, DoorDash, Postmates, Caviar, Seamless, and Eat24. Now Uber Eats and DoorDash are the only two that matter.

Competition like that is good for buyers. It’s brutal for the companies in it. You burn an enormous amount of money on a race that only pays if you’re the only one still standing at the end.

Eventually the survivors stop fighting and start combining. It’s the fastest way to get enough market share and enough leverage to scale toward profitability.

The dbt problem

dbt had a specific version of this. They had built too good of a free product.

The developer experience was excellent. The value was enormous. That’s exactly what made the usual open-core move hard: give the core away, sell an enterprise tier on top.

An enterprise tier has to answer one question. Can this product scale to what the company needs? In practice that means security, things like access controls and audit logging and compliance. It means compute. And it means whatever else a large organization requires before it will run something in production.

Compute was never on the table for dbt. It runs in the customer’s warehouse, on the customer’s bill, and there’s no clean way to take it back.

That left orchestration as the most meaningful thing dbt could sell an enterprise. But orchestration was already its own pillar, with established products serving it. dbt Fusion was an attempt at a better engine underneath. The structural problem stayed. There was no obvious path from a beloved free product to a sustainable enterprise business.

Fivetran had the opposite position. Strong revenue, a large installed base, and the default answer when you ask an enterprise what they pay for data movement. They knew where they stood, and their pricing reflected it. That pricing is the opening the rest of the market walked through. It’s why a value play worked at all.

Put the two together and two of the four pillars sit under one roof. Whatever else you think of the deal, that’s a strong story to tell a buyer.

What the merger meant for my project

I was leading a data orchestration project at Airbyte. The ambition was to eventually compete with Airflow, Dagster, and Prefect. That meant expanding into another pillar of the stack, which was the same instinct behind the Fivetran and dbt merger itself.

After the merger announcement, the future of that project was doomed.

So I spent weeks making a case. Decks, videos, conversations with people across the company. My argument was that the vision could be repointed at a different problem.

What users kept telling me

In the interviews I was running, the same thing kept happening.

Someone would describe a structured data problem, the kind Airbyte was built for. Then, unprompted, they’d mention piles of unstructured data. Contracts, PDFs, internal docs, support tickets. No good way to build pipelines for any of it.

They wanted onboarding that didn’t require walking someone through it. They wanted internal documents that were actually findable. They wanted their AI systems to use their own data.

The orchestration work could serve that. That case didn’t land. Then came layoffs.

Building a RAG is easy

I had already done the interviews. I had already written the argument. I’d talked to enough users to know the problem existed. The only missing piece was someone to go build it.

I’d also underestimated the work.

Building a RAG is easy. I can build one in under a day, and so can you.

I’m going to say it again.

Building a RAG is easy. Building a high quality pipeline that feeds it takes months.

Start with the source. What format is the data actually in? Digital PDFs, scanned PDFs, images, slide decks, HTML, or some combination of all of them.

Then parsing. You convert that unstructured source into a structured representation of itself, usually markdown, because markdown is readable by both humans and models. Converting to markdown is relatively easy. Making sure the markdown is an accurate representation of the original is extremely difficult, especially at scale. This is the hardest step in the process, and everything downstream inherits whatever parsing got wrong. I’ll go into far more detail on this in future posts.

Then chunking, which is where people underestimate the work. You split the parsed document into smaller pieces. Partly because of context rot, partly because if only one section of a contract is relevant there’s no reason to retrieve the whole contract. But deciding where a chunk starts and ends is a judgment call. The only way to know if you got it right is to run it and look at what comes back. That’s experimentation, and experimentation takes time.

Then embedding, the easy one. Turn text into coordinates so a query can find its neighbors.

Around all of that sits the rest. Metadata extraction, enrichment, entities and relationships if you’re building a graph. Which of those you need depends on what you’re building, and your pipeline has to account for it.

Then the question that breaks prototypes. Does it scale? Handling a few hundred PDFs tells you almost nothing about handling a few million. Add scanned documents next to digital ones and the parsing step changes. Add PowerPoints and images and it changes again.

I’ve touched on a lot of these themes in other posts, including Your RAG Retrieval Isn’t Broken. Your Processing Is., RAG Is Dead (And So Is Email Search), and Why I’m Building VectorFlow.

The same problem, one layer up

This looked familiar. Bespoke pipelines, multiple teams, months of work before anything was usable.

That’s where data integration sat before the modern data stack commodified it. Nobody had done the same for unstructured data. Everyone building for AI was back to hand-rolling pipelines, and often didn’t realize that was the expensive part, because the RAG demo worked fine.

That was the thesis behind VectorFlow. Take the complexity out of the pipeline so building one takes days instead of quarters.

I thought that was the hard part. Getting anyone to find it turned out to be harder, which is the next post.

← Back to Home