Can the Modern Data Stack fix private markets' data chaos?

At Opto, we tackle big problems head on. Right now, the private markets industry is facing a data dilemma that’s holding back its growth in the wealth channel.

Why private markets data is different (and fascinating)

Private markets aren’t just illiquid versions of public markets - they’re fundamentally different beasts. I often compare their development to the early days of bond markets: everything started with a prospectus… a PDF. That’s still where most private markets data lives today. But the complexity in processing it has exploded. For example:

  • Schema chaos: Every fund manager structures data differently. There’s no CUSIP-equivalent standardization.
  • Temporal complexity: You need to track not just what changed, but when you knew about it. This is essential for accurate point-in-time analysis.
  • Context matters: In private markets, the story behind the data is as important as the data itself. A 20% IRR means different things in different contexts.
  • Regulatory requirements: We need to move fast - but not break things. In financial services, velocity must be balanced with fiduciary duty and compliance. The question is: can we iterate quickly while maintaining the rigor our market demands?

As private markets make their way into everyday portfolios and retirement savings, transparency isn’t optional anymore. We need infrastructure that can handle this complexity.

Can private markets benefit from the Modern Data Stack?

This is the question we’ve been exploring at Opto. The Modern Data Stack has transformed how tech companies handle data - but financial services is much earlier in that journey.

We have been exploring:

  • The MotherDuck ecosystem: DuckDB as our query engine, DuckLake for S3-based table storage with built-in lineage tracking and time-travel capabilities
  • Dagster: For orchestration with asset-based workflows, partitioning, and event-driven sensors

What we like about MotherDuck

This has rapidly become our new standard. It offers features that deliver operational clarity and rapid experimentation without heavy infrastructure:

  • No Spark. No clusters. No massive machines. Simplicity and speed over complexity - it aligns with our search for fast feedback loops and shorter iteration cycles.
  • Rapid exploration. When discovering new datasets or fine-tuning parsing tools, the iteration speed is astonishing.
  • Built for data engineers. DuckDB’s query engine and DataFrame integration make working with raw Parquet files feel prehistoric.
  • Time-travel and metadata built-in. These capabilities give us lineage, transparency, and reproducibility out of the box.

What we like about Dagster

  • Seamless integrations. MotherDuck, Fivetran, dbt, Airbyte - everything connects cleanly and works together from day one.
  • Software-defined assets. This paradigm shifts accountability: you’re not just ingesting data - you’re delivering a data product.
  • Built-in data reliability. Assets bring lineage tracking, freshness policies, data quality checks, and alerts - right in the orchestration layer.
  • Easy backfills and parallel processing. Dagster handles hundreds of fund manager filings in parallel with strong observability and no manual babysitting.

These tools aren’t just convenient - they enable the architectural principles we need: bi-temporal tracking, schema evolution, and non-destructive curation.

Small Data, big vision

I recently attended Small Data SF, and it was energizing. There’s something powerful about being in a room full of data nerds who get it: the hardest data problems aren’t always “big data” problems - they’re complex data problems. Stay tuned for a full debrief on my experience.

Hearing from data practitioners tackling complex problems across industries made one thing clear: the Modern Data Stack is evolving fast - and in private markets, we see a rare opportunity to lead that process.

We continue to build out our data function. We don’t have all the answers, but we have:

  • A clear vision of what an unopinionated, bi-temporal data infrastructure looks like
  • A willingness to bring modern tooling to a traditional industry
  • A commitment to solving problems at the intersection of “regulatory compliance” and “modern data practices”
  • A path toward building a comprehensive and high-quality private markets database that will serve as the backbone for making informed investment decisions and growing our business

We’re just getting started - building the next generation of private markets data infrastructure and learning fast. It’s messy, it’s challenging, and it’s exactly where innovation thrives.

For disclaimers, visit https://www.optoinvest.com/disclaimers.