In search of a data tribe.
Small Data SF: A conference debrief
At Opto, we’re encouraged to take initiative, try new things, and step outside of our comfort zones. When MotherDuck announced the second edition of Small Data SF in early November, I jumped at the opportunity to attend.
More than just swag
Yes, MotherDuck’s branding was fantastic (the swag delivered), but I wanted to go deeper. After three months as an early adopter of DuckLake, I’d already been in the trenches with their team—sharing what worked, flagging what didn’t, and learning alongside people tackling some truly hard data problems. That groundwork made the in-person experience special. When I arrived, multiple MotherDuck team members immediately recognized me: “You’re Lucie—we see you on Slack!” There’s something powerful about that recognition. It transformed a professional conference into a community gathering immediately.
Day zero: Women in Data dinner
I was humbled to be invited to a special pre-conference dinner focused on women in data. This wasn’t just networking. It was an energizing, honest exchange about the realities of building data teams in complex industries. My objective for this dinner was clear. I was going to leave with two to three solid contacts. I left with far more—meaningful conversations about the trade-offs leaders face between chasing hype and seizing genuine opportunities.
Day one: workshops that deliver
- Dagster. I wanted to solidify my learning and connect with the team. What impressed me most? My colleague Victor, who had never worked with an orchestrator, got a complex pipeline running by the end of the session. This reinforced my belief that the modern data stack simplifies workflows, letting engineers focus on data quality while delegating infrastructure complexity to purpose-built tools.
- Bem’s Unstructured-to-Structured Data. Bem is building AI that converts unstructured data into structured formats. Right up my alley. We tested their API live, converting files in real-time. Their UI was slick and impressive. Seeing tools like this mature makes me optimistic about tackling private markets’ PDF-first reality.
Day two: the main event
The conference opened with a line up of impressive practitioners sharing real-world experiences. One moment stuck with me. Jordan Tigani told the origin story of MotherDuck—how it started almost as a joke, owning the term “small data” and inviting people to laugh with them. He reminded us that not long ago, if you weren’t launching Spark clusters or parallelizing processes across massive machines, you weren’t considered a “real” data engineer. The small data movement is reclaiming that narrative. You can solve complex problems on a single machine. You can prioritize simplicity over infrastructure theatrics. And yes, you are doing serious data engineering.
Why this matters for Opto
At Opto, we don’t shy away from complexity. Solving private markets’ data challenges requires more than great technology. It requires building a community of people who care about the same problems.
Private markets are at an inflection point. As alternative investments move into everyday portfolios and retirement savings, transparency and data quality are no longer optional. We need infrastructure that can manage schema chaos, temporal complexity, unstructured data, and regulatory nuance, while still moving fast.
Small Data SF was a reminder that we’re not alone. There’s a growing community of practitioners who believe you can have speed, simplicity, and sophistication, and that the hardest problems aren’t always “big data” problems, but rather complex data problems.
Private markets, we’re coming for you. With the modern data stack and a community that gets it, we’re building the next generation of financial data infrastructure. And we’re just getting started.
For disclaimers, visit https://www.optoinvest.com/disclaimers.