The Clock Is a Server

At ten to three in the morning on July 8, 2026, Telstra's network forgot what time it was. One of the carrier's time servers, the boxes that tell the rest of the network what time it is, had reset its calendar to 2006, and the bad time was propagating outward through the synchronization layer. By breakfast, 8.8 million Telstra customers were on services that were degraded or dead. Victoria's V/Line regional railway stopped running trains. National freight operators suspended movements as a safety precaution. Tyro, the payments company that handles card terminals for about 80,000 Australian businesses, started dropping transactions at shop counters. 612 triple zero emergency calls failed on Telstra's network, more than 430 of them without reaching anyone at all, and every one ended with a welfare check on the person who had tried to call for help. Telstra had the time fault fixed by about 4 p.m., some thirteen hours in, and knock-on problems ran into the next day.

Nobody attacked anything at any stage.

The independent review Telstra released on September 1, 2026, found that a power supply swap on a Melbourne time server kicked it back 1,024 weeks, to 2006, the classic GPS week-number rollover. (This is an excellent time to remind everyone to read Falsehoods Programmers Believe about Time) The vendor had issued advisories about that bug as early as November 2000 and Telstra's own technicians logged it in September 2022 and decided it didn't apply to them, because the patch fixed a feature they weren't using. Then an undocumented design change in October 2025 started using it, which, of course, nobody wrote down, so in January 2026, with the bug now very much live, they looked at the patch again and passed again. The alarms that would have flagged the timing drift were only watched during business hours, by a limited number of people, and two of Telstra's key engineers were on mandatory leave that night. The review's overarching finding was that Telstra never treated its timing system as a critical capability. CEO Vicki Brady's summary: "That is a miss on our side." This was a known defect in a component nobody treated as critical, because time does not feel like a component until it stops.

As I write this in early October, ACMA is still investigating, with fines of up to A$30 million on the table. On September 21 a Senate committee recommended taking Triple Zero away from Telstra and handing it to a new public body, and on October 1 the three big carriers switched on roaming onto each other's networks for when a disaster takes one of them down. Bushfires and floods, mind you. Nobody has a roaming plan for a clock that thinks it's 2006.

To see why one bad calendar could do that, start with the radio network. In the time-division duplex modes used by modern LTE and 5G networks, the tower and the handset share one slice of spectrum by transmitting in alternating slots, and the specifications require every cell to hold its slot boundaries within roughly 1.5 microseconds of a common reference, so no two neighbors drift more than about 3 microseconds apart. Beyond that tolerance, adjacent towers begin transmitting over each other's frames. Carriers therefore operate a dedicated distribution system for time itself, in which receivers discipline themselves against atomic clocks in orbit, the Precision Time Protocol carries the reference across the backhaul, and holdover oscillators at each site keep an approximation running whenever the upstream reference goes quiet. Time on a network like Telstra's is produced in a small number of places and consumed by tens of thousands of sites, and in July the production run was defective.

Distributed systems people have known the underlying truth for decades, and everyone else keeps rediscovering it at three in the morning. Computers cannot vote their way to the correct time, because every voting protocol needs timeouts and every timeout needs a clock, so Leslie Lamport's famous 1978 move was to give up on clocks and derive ordering from messages instead. Google went the opposite direction with Spanner's TrueTime, installing GPS receivers and atomic clocks in its data centers and having the system report an uncertainty interval and wait it out rather than pretend the time is exact. Opposite philosophies, same admission: somewhere in every distributed system sits a reference authority everything else syncs to. You do not get to not have one. The design choices are how many you have, who operates them, and how far a lie travels when one of them lies. Most of the internet answers those questions with "a handful," "volunteers and hyperscalers," and "we'd rather not think about it," courtesy of NTP.

On the night of June 30, 2012, a leap second was inserted into civil time to keep atomic clocks aligned with the earth's rotation, and a bug in the Linux kernel's handling of the extra second sent processes on servers around the world into high-CPU spin loops. Reddit went down for a while. Mozilla, LinkedIn, Yelp, and Gawker all reported problems that night. The Amadeus airline reservation system failed in Australia, and Qantas staff checked passengers in by hand for hours until it recovered. Four years later, in January 2016, the US Air Force decommissioned a GPS satellite designated SVN-23, and an erroneous ground-software upload put a 13-microsecond error into the time broadcast by 15 satellites, about half the constellation. Over the following hours, the BBC's digital DAB radio listeners lost reception because the transmitters took GPS as their timing reference, and one timing-equipment user logged nearly 2,500 alarms before the problem cleared.

We did solve this once, at continental scale, and the people who solved it ran trains. On November 18, 1883, the North American railroads imposed five time zones on the continent by private agreement, thirty-five years before Congress ratified the arrangement in the Standard Time Act. And when synchronization still failed, as it did near Kipton, Ohio in 1891, where a crew member's watch ran slow and two trains met on the same track, the response was Webb C. Ball's watch inspection regime: approved movements, thirty seconds a week of allowed drift, inspections on a schedule, paperwork per timepiece. The reference was standardized, and then every consumer of it was audited as an independent clock, so a watch could still fail but a defect could not propagate. Telstra's design inverts that: centralize the reference so everything downstream can be a dumb consumer, and one bad clock becomes everyone's outage at the same instant.

Time is upstream of everything, so a time failure surfaces downstream everywhere, wearing everyone else's costume. On July 8 it looked like a train problem, a payments problem, and an emergency-line problem, and the fix took half a day at one of the most sophisticated operators in the hemisphere, because you cannot debug a distributed system when the broken thing is the thing you use to order events. Every log line on every node carried a timestamp from the same poisoned well.

So, a suggestion, and it is the same one the railroads landed on after Kipton. Go find out where your time comes from. Not conceptually; specifically. How many independent sources, owned by whom, checked by what, and what your systems do in the hours after the holdover oscillators start to drift. If the answer is "one vendor's servers in two cities," you do not have a clock. You have a subscription to someone else's, and on July 8 all of Australia got to read the terms.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

NOTE: I'm currently writing a book based on what I have seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. I'd love to hear your thoughts!