Two Hours Behind

On December 24, 1814, British and American negotiators signed the Treaty of Ghent. The document then got put on a boat. On January 8, 1815, fifteen days after the signing, British troops attacked Andrew Jackson's defenses outside New Orleans. The American Battlefield Trust counts 2,034 British casualties, including 285 killed, against 71 American casualties.

The familiar version of this story says they fought a battle after the war was over. They didn't. The treaty still needed American ratification. The Senate approved it on February 16, and the exchange of ratifications on February 17 ended the war. On January 8, the armies still had orders to fight. An agreement had been reached on the other side of the Atlantic, but the process needed to make it effective had not finished.

Nobody blames those commanders for not having a telegraph. I am less sympathetic when we build a fast network and then deliberately leave the data waiting at every stop.

Ask whoever owns your real-time dashboard how long an event takes to become a pixel. The query latency is only part of the answer. In front of it sit a scrape interval, a transport hop, a queue, an ingest batch, a transform job, a materialization schedule, and a cache TTL nobody has looked at since it shipped. A dashboard can load quickly and show you something that happened two hours ago.

Count the waiting

Take a deliberately simple example. Collection runs every 60 seconds. Transport takes two seconds. Ingest flushes every 30 seconds. Enrichment runs every two minutes. After the enriched data reaches the alert evaluator, the condition must remain true for ten minutes before it fires. That last delay might be entirely reasonable after last year's on-call burnout.

For an event that just misses each scheduled step, the budget is 60 + 2 + 30 + 120 + 600 = 812 seconds. Thirteen minutes and thirty-two seconds, before notification delivery or human response. That's an illustrative sum of maximum scheduled waits, not a measurement or a prediction that every event takes that long. Some stages may line up more favorably. Backlogs and retries can make the trip longer.

The alert rule needs particular care. A five-minute lookback window does not automatically mean a five-minute wait. A pending duration does: in Prometheus, a for: 10m clause requires the condition to stay active through evaluations for ten minutes before the alert fires. Evaluation cadence adds its own delay. Read the rule you deployed before adding numbers to the budget.

If the promise is to act within five minutes of the event, this design cannot meet it. If the promise starts at detection, write down what counts as detection. Starting the clock after ten minutes of waiting makes the metric easier to satisfy without getting anyone help sooner.

Each team may have made a sensible choice. Bigger batches reduced storage overhead. Less frequent enrichment saved compute. A longer pending period cut noisy pages. The person on call gets the sum of those decisions. You have a fucking architecture problem wearing a dashboard as a disguise, and the postmortem will blame the dashboard.

Buying more query compute will not shorten a configured ten-minute hold. Someone has to decide whether that wait belongs in this path.

Give each signal a budget

Industrial systems make this distinction explicit because a late response can damage equipment. Sielco's comparison of SCADA and industrial IoT separates production-critical supervision from supplementary sensing and analytics. A plant-wide energy dashboard and a machine's protective control loop have different jobs, even when they use readings from the same equipment.

That is a useful discipline to bring into a dashboard review. Decide how late each answer can be before choosing its route. A maintenance trend may tolerate minutes. A protective response may need a dedicated local control system, with requirements a general-purpose telemetry pipeline cannot satisfy. Moving an analytics rule to an edge server does not turn it into a safety system.

You also need to decide what information can survive aggregation. An average can be useful for a trend and useless for detecting a brief spike. Keep the resolution the decision needs, whether you evaluate the signal locally or centrally.

Keep the center where it helps

I have architected plenty of centralized telemetry and would again. The mid-incident questions are often cross-system questions. Correlation, retention, and the auditor's copy are good reasons to keep central processing. A centralized streaming system can also handle fast reactions when its transport and processing fit the budget.

When they don't, move the fast rules earlier in the pipeline, sometimes to the source. Filter, enrich, classify, and evaluate what can be decided there, then send the center what it needs. Some decisions need the wider context. Others should not wait for a warehouse refresh to discover that one machine crossed a threshold.

In March I wrote about the time value of data. The practical follow-through is to measure where that time goes. Keep the event timestamp through the pipeline. Record when the data arrives, when the rule evaluates, when the notification goes out, and when someone acts. Account for clock skew before treating differences between machines' timestamps as precise measurements.

Then test the whole path under load, including the batch boundaries. Show the age of the data on the dashboard alongside the result. Give someone ownership of the event-to-action budget, because owning the query is only a small part of that job.

Before the next incident, put your promised response time beside a measured trip through the system. If the waiting alone exceeds the promise, you have a specific configuration or architecture decision to change.

Want to run the filtering and the fast decisions where the data is born, instead of two hours downstream of it? Check out Expanso. Or don't. Who am I to tell you what to do.

NOTE: I'm currently writing a book based on what I have seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. I'd love to hear your thoughts!