data pipelines - Distributed Thoughts

Distributed Thoughts

Sign in Subscribe

data pipelines

A collection of 12 posts

The Inference Inversion

distributed computing

The Inference Inversion

For the past three years, the AI industry has operated under a simple assumption: more centralized compute solves everything. Bigger clusters. Bigger data centers. Bigger power contracts. The logic was intuitive and, for training workloads, largely correct. Training a frontier model requires tightly coupled GPU clusters with high-bandwidth interconnects, and

The Time Value of Data

The Time Value of Data

Your data is rotting. Not metaphorically. Not in some hand-wavy "data quality matters" sense. Your data is losing measurable economic value with every hour it sits unprocessed, and the infrastructure you built to manage it has no concept of that loss. No depreciation schedule. No freshness SLOs. No

The Missing Part of the Pipeline

The Missing Part of the Pipeline

Imagine a customer support chatbot that confidently tells a user they have 365 days to return a product. The actual policy is 30 days. Every system metric is green: P95 latency at 142ms, throughput at 1.2k requests per second, eval accuracy at 94.2%, error rate at 0.02%

NVIDIA Just Showed You the $249 Future. Nobody Built the Layer That Makes It Work.

NVIDIA Just Showed You the $249 Future. Nobody Built the Layer That Makes It Work.

Well... not NOBODY :)

The Bronze Tier is Far From Gold: Avoiding Toxic Assets (And Other Medallion Architecture Lies)

medallion architecture

The Bronze Tier is Far From Gold: Avoiding Toxic Assets (And Other Medallion Architecture Lies)

LET'S SAY you're a hot new drone company. Doing, I don't know, crowd counting at festivals. One 4K camera generates roughly 45 gigabytes of footage per hour. That’s your baseline. Add additional sensors (battery, motor, LiDAR for terrain mapping, thermal imaging, and flight

The History of Expanso (Part 2): Our Core Tenets

The below is a continuation of the series on the history of Expanso. Today, we're talking about the tenets of Expanso. Read the whole series starting from Part 1. The Four Tenets of Sane Data Pipelines In the last post, I explained the origin of Expanso: hitting the

The History of Expanso: It's About the Network (It Always Has Been)

Welcome to our series on the history of Expanso. The Network is the Problem: How Expanso Began The first time the scope of the problem facing tech was super clear, I was at Protocol Labs. We had IPFS storing and serving data at exabyte scale. The storage worked. The catalog

The Distributed Data Dilemma: Kubernetes' Unsolved Puzzle

The Distributed Data Dilemma: Kubernetes' Unsolved Puzzle It was a question I heard constantly in the years after Kubernetes launched: "Can we run our database on Kubernetes?" The answer was always, "Yes, but it's complicated." It wasn't complicated because of Kubernetes.

From Kubeflow to Real-World ML: Why Data Locality Matters Just as Much as Compute

From Kubeflow to Real-World ML: Why Data Locality Matters More Than Compute When my co-founders, Jeremy Lewi, Vishnu Kannan, and I started Kubeflow back in 2017, we were trying to solve what felt like the biggest problem in machine learning. Brilliant data scientists would craft elegant models on their laptops,

The Myth of Portability: Helm and Kubernetes and the Data Pipeline Problem

The Myth of Portability: Helm and Kubernetes and the Data Pipeline Problem I spent years helping companies migrate from bash scripts to Chef, and later to containers. The conversation always started the same way: "We want modern infrastructure, but it has to work with what we have." The

Unlocking Reliability: Why Data Pipelines Need Declarative Deployment & GitOps

Unlocking Reliability: Why Data Pipelines Need Declarative Deployment & GitOps You know the feeling: your data pipeline worked perfectly last week, and now it's throwing cryptic errors. The logs don't help. The documentation is outdated. Nobody's sure who last touched the transformation logic or

Orchestration Allows Microservices to Be Unreliable (That's a Good Thing)

Orchestration Allows Microservices to Be Unreliable (And That's a Good Thing) One of the first features I wanted to build for Kubernetes was service workflows: Service A starts, then B, then C. If B fails, A should know, and C shouldn't panic. Services need to know