data pipelines - Distributed Thoughts

Distributed Thoughts

Sign in Subscribe

data pipelines

A collection of 8 posts

The Bronze Tier is Far From Gold: Avoiding Toxic Assets (And Other Medallion Architecture Lies)

medallion architecture

The Bronze Tier is Far From Gold: Avoiding Toxic Assets (And Other Medallion Architecture Lies)

LET'S SAY you're a hot new drone company. Doing, I don't know, crowd counting at festivals. One 4K camera generates roughly 45 gigabytes of footage per hour. That’s your baseline. Add additional sensors (battery, motor, LiDAR for terrain mapping, thermal imaging, and flight

The History of Expanso (Part 2): Our Core Tenets

The below is a continuation of the series on the history of Expanso. Today, we're talking about the tenets of Expanso. Read the whole series starting from Part 1. The Four Tenets of Sane Data Pipelines In the last post, I explained the origin of Expanso: hitting the

The History of Expanso: It's About the Network (It Always Has Been)

Welcome to our series on the history of Expanso. The Network is the Problem: How Expanso Began The first time the scope of the problem facing tech was super clear, I was at Protocol Labs. We had IPFS storing and serving data at exabyte scale. The storage worked. The catalog

The Distributed Data Dilemma: Kubernetes' Unsolved Puzzle

The Distributed Data Dilemma: Kubernetes' Unsolved Puzzle It was a question I heard constantly in the years after Kubernetes launched: "Can we run our database on Kubernetes?" The answer was always, "Yes, but it's complicated." It wasn't complicated because of Kubernetes.

From Kubeflow to Real-World ML: Why Data Locality Matters Just as Much as Compute

From Kubeflow to Real-World ML: Why Data Locality Matters More Than Compute When my co-founders, Jeremy Lewi, Vishnu Kannan, and I started Kubeflow back in 2017, we were trying to solve what felt like the biggest problem in machine learning. Brilliant data scientists would craft elegant models on their laptops,

The Myth of Portability: Helm and Kubernetes and the Data Pipeline Problem

The Myth of Portability: Helm and Kubernetes and the Data Pipeline Problem I spent years helping companies migrate from bash scripts to Chef, and later to containers. The conversation always started the same way: "We want modern infrastructure, but it has to work with what we have." The

Unlocking Reliability: Why Data Pipelines Need Declarative Deployment & GitOps

Unlocking Reliability: Why Data Pipelines Need Declarative Deployment & GitOps You know the feeling: your data pipeline worked perfectly last week, and now it's throwing cryptic errors. The logs don't help. The documentation is outdated. Nobody's sure who last touched the transformation logic or

Orchestration Allows Microservices to Be Unreliable (That's a Good Thing)

Orchestration Allows Microservices to Be Unreliable (And That's a Good Thing) One of the first features I wanted to build for Kubernetes was service workflows: Service A starts, then B, then C. If B fails, A should know, and C shouldn't panic. Services need to know