It Worked Before We Understood Why

There is a critical hunk of metal inside a jet engine that I think is a wonderful analog to how we should think about AI.

In the hottest part of the turbine, gas can be flowing past the blades at temperatures above the melting point of the alloy they are made from. Those blades are also spinning, carrying enormous loads, and expected to keep doing this while the people in the cabin argue about whether the person in front of them should be allowed to recline.

This seems like a fairly aggressive thing to ask of a piece of metal.

Veritasium's deep dive into jet engines is wonderful on the engineering involved. The survival trick is a combination: nickel superalloys, protective coatings, and cooling air routed through and around the blades. The gas can be hotter than the metal's melting point because the metal itself is kept cooler. Rolls-Royce describes gas temperatures around 1,700°C, several hundred degrees above the blades' melting point. Even with cooling, the material has an big, big job.

Part of what makes these alloys work is a microscopic structure called gamma prime. Tiny particles, often shaped like cubes, sit inside the nickel matrix and interfere with the movement of dislocations, the defects that allow a crystal to deform. Under certain conditions, heating actually makes those dislocations lock up more readily. Over a limited temperature range, the material can become harder to deform as it gets hotter. This is not unlimited, and it is certainly not permission to turn off the cooling, but it is a lovely reversal of what most of us expect heat to do to metal. The microscopy alone is worth rabbit holing on.

The fascinating part of it was the order in which things happened. Useful alloys came before a complete explanation of their behavior. Engineers developed materials, measured their properties, changed their composition and processing, and learned which ones survived. The microscopic explanation was developed alongside and after that work; early nickel alloys were being developed in the 1930s and 1940s; the detailed account of what was happening inside them took longer.

They knew something was consequential before they understood it.

I think we are having a surprisingly hard time allowing that distinction in AI.

Around the same time, I watched Grant Sanderson and Steven Strogatz discuss what AI means for mathematics. There is a section, starting around 33:55, on whether understanding is overrated, which is an entertaining question to hear in a conversation involving the creator of 3Blue1Brown. Making mathematics understandable is pretty much the man's entire thing.

The question I was thinking about was how much do we value understanding for itself and how much because it has been such a powerful way to get something done?

For a mathematician, understanding may very reasonably be the point. For the passenger in 23B, there is a more immediate requirement, which is not falling out of the air.

Of course, humans have always made useful things before fully explaining them. Fermentation predates microbiology by a fairly substantial margin. We did not wait for a molecular account of bread before opening bakeries. AI did not invent the separation between a useful result and an explanation we find satisfying. It is making that separation show up in work where a lot of us had assumed our understanding was indispensable.

That is uncomfortable. I like understanding things. Quite a lot. A good explanation lets you transfer an idea, spot a mistake, and predict what will happen somewhere you have not tested yet. It is enormously valuable. But the feeling of understanding a system and the evidence that it will do its job are different things, and I think we sometimes let the first stand in for the second.

An explanation can be convincing and wrong. A result can be correct even when the process that produced it is opaque. Neither observation tells you, by itself, whether it has value.

The turbine blade is useful here because nobody has to settle a philosophical argument before deciding what to measure. You need to know how it behaves under heat and load, how that behavior changes with time, which defects matter, and when the part has to come out of service. Incomplete microscopic understanding does not eliminate that work. It makes that work more consequential.

And this is where the analogy gets demanding. A turbine blade operates inside a comparatively constrained physical environment. A general-purpose model can encounter a new kind of request, retrieve hostile instructions, or be given access to a tool its designers never anticipated. There is no universal test suite that turns all of that into a certified component. If we want to borrow aviation's confidence, we have to borrow its discipline about the conditions under which that confidence applies.

What interests me, though, is why this feels so personal.

In the video, they talk about the particular pleasure in finally understanding something that has been bothering you when a collection of disconnected facts suddenly becomes a thing you can turn around in your head. You can explain it to somebody else. You can imagine what would happen if one piece changed. The world has become a little less mysterious, and you have become a little more capable of living in it. I don't think we should be embarrassed about wanting that or pretend that getting an answer from a machine provides the same experience.

But I do wonder how much of our discomfort comes from the possibility that the world can become more manageable without becoming more understandable to us. We have spent a long time occupying both roles: the sentient beings who want things to happen and the sentient beings whose understanding makes them happen. It is deeply unsettling to discover that we might get more of the first while being less necessary to the second.

If a machine finds something useful that I cannot explain, my inability to explain it does not make it less useful. It does, however, leave me with a choice about what evidence I would need before depending on it. And that is a harder question than whether its explanation sounds like something a smart person would say.

We already live with an enormous amount of borrowed understanding. I cannot personally explain everything (anything?) about keeping an airplane in the sky, and neither can any one of the people who built it. My confidence rests on other people knowing different things, checking one another's work, recording what went wrong, and having reasons to take those failures seriously. Somewhere in that arrangement, knowledge that no individual possesses becomes something all of us can use.

AI puts pressure on that arrangement because some of the missing understanding may not belong to anyone. There may be no expert down the hall who can sit me down and explain why this particular answer emerged. That is a real difference. It means we have to be much more explicit about what we have observed, what we can establish independently, and what we are merely hoping will continue to work.

It also forces a question we tend to skip when we are impressed by a result: useful to whom?

The passenger wants to arrive safely. The airline wants to operate economically. The engineer wants a component that survives its specified conditions. Even in that relatively concrete example, “it works” contains a collection of human purposes. Move into education, relationships, or the way we organize a society, and agreeing on what counts as success becomes a substantial part of the work. A system can be exceptionally good at producing the thing we asked for while leaving us to discover that we asked for the wrong thing.

Understanding matters here in a different way. We need enough understanding of ourselves to decide which outcomes are worth pursuing, what we are willing to risk, and whose experience counts when we decide whether something succeeded. Greater capability makes those questions more consequential. It does not answer them.

And understanding can still be worth pursuing even when it is no longer required to obtain a result. I would still want to know why those tiny cubes make a turbine blade stronger if every flight I ever took were guaranteed to land safely. There is something wonderful down there, and I want to see it. The practical usefulness of curiosity has helped justify it for a long time, but I would hate to discover that we only valued curiosity because it paid.

Maybe that is part of what we are being asked to get comfortable with: understanding as something we love, sometimes something we need, without insisting that it must precede every useful thing we allow into the world.

That requires humility in both directions. We have to be willing to accept evidence for something we cannot yet explain and equally willing to admit when the evidence is too weak, when the consequences are too large, or when we have confused getting an answer with knowing what to do. We remain responsible for the decision to trust it, including the consequences for people who never got a vote.

The engineers working on those alloys had somewhere they wanted to go and an incomplete account of what would get them there. They measured, tested, learned, and kept investigating. I find that a much more human story than the idea that progress happens only after somebody has understood everything.

We have always lived between what we can explain and what we are prepared to try. AI is making that space larger, and we will have to learn how to inhabit it carefully.

There are still people on the plane.

Further reading: NASA's 1972 historical review of high-temperature alloys, Superalloys: A Primer and History, and Cambridge's introduction to turbine-blade coatings.

Want to run the filtering and the fast decisions where the data is born, instead of two hours downstream of it? Check out Expanso. Or don't. Who am I to tell you what to do.

NOTE: I'm currently writing a book based on what I have seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. I'd love to hear your thoughts!