Understanding comes before modeling: a data engineer's philosophy for the age of AI extraction

A data engineer’s playbook for the age of AI extraction

AI extraction is table stakes now. Point a model at a prospectus, an invoice, a claim, or a contract, ask for a table, and you get one. Work that used to take an analyst a day takes about a minute.

That’s a genuine game changer, especially if you’ve read way too many bond prospectuses in ugly PDFs. It’s also a trap. In my experience, the teams that get the most out of AI are the ones that understood their data before they modeled it. This post is about what that looks like in practice.

It started with a missing price

I started my career as a data analyst. Part of my job was delivering reports at month end with accurate data, so quality control was a big focus. I got good at it quickly, because I noticed that data errors are almost never random. If you find one missing value, there’s usually a whole bunch of rows with the same problem, caused by the same thing. So you find the pattern, fix the batch, move on.

I got a little obsessed with one problem in particular: missing prices. And every time, it was the same story. A script had stopped loading data, and nobody knew.

That’s what pulled me into engineering. I was tired of losing my analysis time to debugging pipelines, so I went and read the code.

To my surprise, the scripts weren’t wrong. They matched the requirements they were given at the time they were written. Nobody ever went back and asked, “is what we understood back then still true?” We just kept adding on top of it.

But data isn’t static. The world evolves, and so does the data.

That was my first real lesson: leave room for the data to evolve. A model is a snapshot of what you understood on the day you built it. Lock it in too early, and you’re building on something that’s already going stale. So you understand first, by looking at the data again and again and again. And you model later.

Why this, why now

For the past twelve months at Opto, we’ve been focused on the data model. Not the tools, not the dashboards. The model. Three reasons:

  • There’s no clear winner for private markets data. Nobody has handed us “the” model, so we have to define it ourselves.
  • Many flavors of the same thing exist. Two things that sound identical can behave completely differently.
  • We hope to sell this data one day. So good enough isn’t enough.

One note before going further. This post is about techniques, not a blueprint. Our stack mostly reflects what was already in place, what let us move fast, and what works for an SEC-regulated company. Other technologies could do the job just as well.

Two ways to get this wrong

Here’s what most people do with AI extraction. They take their data model, their final tables, and tell the LLM, “here’s my schema, fill it in.” A document goes in, a clean row comes out, straight into the model.

It feels efficient. It’s also a trap, because your extraction only has room for what you already understand. Anything that doesn’t fit gets dropped, or forced into the closest field. Then next quarter the requirements change, they always do, and you’re re-extracting the same documents, if you even kept them.

So people overcorrect: “Fine, I’ll extract everything.” That’s the opposite trap. Every document becomes its own table, with everything in it, and nothing is normalized.

Yes, I’m of the opinion that normalization is still a thing. It just means making the same thing look the same. If nobody normalizes the data, every question you ask turns into a cleanup project first. You don’t have a story, you have a pile. And you burn unnecessary tokens, again.

Both traps are the same mistake: modeling before understanding. In the first, you decided too early what mattered. In the second, you never decided anything. AI makes it a thousand times faster, and the output looks clean so nobody questions it until something depends on it.

The funnel: raw, transform, model

I like to think of a pipeline as a giant funnel, where you learn a little more about the data at every stage.

Some of you may be thinking, “you’re describing the medallion approach.” True and not true. Yes, it looks a lot like bronze, silver, gold (I just like to be different, so I named them differently). But it serves a different stage of your data’s life. This is what you do before your data model is set.

Raw. You keep the source exactly as it arrived. No parsing, no cleanup. It’s boring, and that’s the point. It’s your insurance: whatever you mess up later, you come back here and start over. In an AI world, one more thing belongs here: what the AI extracted. That output, in table form, exactly as it came back, is raw too.

Transform. Now you have raw data you can actually look at. Once upon a time, this is where Jupyter notebooks were the game changer. Load a million-row dataframe, find where column A is empty, notice that half the dataframe has it, then notice it’s the same half where column B says “quarterly.” Just like in quality control, it’s not random. That’s how we used to discover things.

Now, as long as you have enough data volume, you can simply ask an AI: “Tell me about the content of my data. What patterns do you find?” This is where you bucket by theme, by source, by the shape of the values. You normalize here too. You’re learning what’s actually in there, one step at a time.

Model. The model comes last. Now you resolve entities and map everything to the shape your business runs on. And since every layer can be re-run on its own, a change in business logic means redoing the model, and a change in extraction means redoing the layer before it. Three types of mistakes, three different undo buttons. Decide last, redo anytime.

The field that wasn’t one value

A few weeks ago this approach saved us. We had a field built to hold one value, which sounds logical. I was migrating data to a local version of our new system, doing some extra testing, and realized the field was empty for a lot of rows. I asked AI to look at the source and tell me why.

It was a tiered schedule: a different number depending on a condition somewhere else in the document. It didn’t fit one field, one value.

BEFORE: one field, one number

  rate = ?


AFTER: one row per tier

  condition             | rate
  ----------------------+------
  under 10              | 5%
  10 and over           | 3%

If we had gone straight to AI, I can tell you exactly what it would have said: “Have you tried converting that column from string to JSONB?” Then your AI session tells you it’s an easy fix, tests it, and ten minutes later everybody’s happy. It would also have been the most expensive way to hide a tiered schedule in a text blob. Stored data has to be intuitively queryable.

Instead, because the raw and transform layers were right there, we had room to try. We could look at it, test a few shapes, and find the modeling that fit best. The decision didn’t happen by accident. It was researched.

This isn’t a finance thing. A liability clause with conditions. A dosage that depends on body weight. A shipping rate that depends on the zone. Every domain has a value that’s usually one number, until one document shows up where it isn’t. The funnel gives you somewhere to be surprised safely, instead of in production.

And that’s the first takeaway: nobody decides what the data means all at once. The opinion builds one layer at a time, and something has earned its place by the time it reaches the model.

Lineage: one fact, one source

The funnel tells you what happens to data as it moves through. But in real life you don’t have one source. You have many, and they don’t always agree.

The rule we follow is one fact, one source. For any piece of data, we can always answer one question: who told us this? If source A says a price is 10 and source B says 12, we keep both, each with its own source tag.

That one rule gives us three things.

Who can see what. Every fact has a source tag, so access is really a question about the source. That matters a lot if you hope to sell your data one day. It also matters for a data agent that should only see some of it.

Fixing a wrong match. Every time something new comes in, we ask: do we already know this entity, or is it new? Sometimes we get it wrong. We link two entities that aren’t the same, or keep apart two that are. Because we kept exactly what each source sent, we don’t guess and we don’t edit by hand. We unwind it and replay it. Nothing else gets touched.

Deleting safely. You can’t remove something if other things are built on it. Say one source asks us to delete its data. We know exactly which facts carry its source tag. An entity that only that source had? Safe to remove. An entity that three other sources also have? It stays, and we just take that one source tag off it.

And when a new source comes in, it just adds its own rows, with its own source tag. Nothing that’s already there changes. So keep every source’s claim, with its source tag, and you can share it, fix it, and remove it without breaking anything else.

What this structure lets you do

Here’s where AI gets interesting. Not one big, clever agent that does everything, but a few small ones, each with one job on one layer.

Quick definition, because people mean different things by “agent.” I mean a model with a playbook and tools that works through a task. It doesn’t have to run unattended. Mine mostly don’t.

The gap agent works on the raw and transform layers. Most people build an agent to load data: you give it your data model and say “fill this in,” so when something doesn’t fit, it forces it. The gap agent has the opposite goal. It doesn’t load anything. It reads the data, compares it to the data model, and finds what the model can’t hold. It follows a set of steps, it has the knowledge it needs, and it hands back a clear gap plus an example of how to model it, with a before and after. The tiered schedule above is exactly the kind of thing it finds. A person decides what to do next.

The QC agent works on the transform layer and catches problems before they reach the model. Remember where I started? This is my bread and butter. It checks the data against a list of known ways it can go wrong, like a net number that’s bigger than the gross number, or a negative rate. For each issue it asks one question: can I decide this, or not?

If it’s clearly wrong and it can fix it, it does. But it doesn’t edit anything. It loads the fix as its own source, so the agent’s opinion is just one more data point, labeled as the agent. If it’s not sure, it fixes nothing and hands it to a person.

Here’s an example. It flagged two records as a probable naming mistake, a classic data error. It turned out to be a real acquisition: one company bought another and kept the brand name. The agent couldn’t tell from the data, so it asked instead of fixing it. If we had let it auto-correct, we would have erased a fact.

The intelligence agent works on the data model itself, which is a map of things and how they connect. For us, that’s typically a financial graph. Its job is to follow those connections and answer questions.

None of these works because of a clever AI model. Each has a small job on a layer we understand, and each knows when to stop and hand it to a person. An agent on a half-understood setup just guesses faster.

The real payoff

All of this sounds slow, and on day one it is. But it’s how you scale, because it compounds. Each time a document surprised us, the funnel absorbed it. We didn’t rewrite anything, we extended the model. A new source adds its own rows and changes nothing that’s already there. A new kind of document either fits, or it shows up as a gap. Every surprise gets cheaper than the last one.

One last note

Understanding feels slow and overwhelming on day one. You don’t know where to start. But it gets easier and faster as you lean into it. And on day 300, you’re faster than ever.


For disclaimers, visit https://www.optoinvest.com/disclaimers.