Manufactured Unmeasurability

In the North Atlantic, off the coast of Greenland, a line of instruments has been hanging in the water for years. They are moorings — anchored sensors that measure the temperature, salinity, and speed of the deep current that carries warm water north and keeps the climate of Europe and the eastern seaboard inside the band we are used to. The current is the Atlantic Meridional Overturning Circulation, and a growing body of work suggests it is weakening toward a threshold past which it does not gradually recover.AMOC observing^[1] This year, the agency that maintains the Irminger Sea moorings began pulling them from the water. The strangest part is that the money was there. The administration had twice proposed gutting the program, and twice Congress restored the funding — and the decommissioning went ahead anyway, set in motion by the threat and too far along to stop once the threat had passed. The instrument that would tell us how fast we are approaching the cliff is being removed as we approach the cliff, and not even a won budget fight could keep it in the water.

The same week, two other instruments came out of the water, in two other domains, for two completely different reasons. Anthropic released Fable 5 and built it to route its highest-risk requests — the cyber and bio queries that sit near the dangerous frontier — away from direct handling and into a separate, more guarded model.Anthropic^[2] This is clean safety engineering. It is also a withdrawal of direct observability: the most consequential behavior now happens one inferential step removed from the operator who would otherwise be watching it. And in the Gulf of Oman, after an Army helicopter went down near the Strait of Hormuz, an uncrewed surface vessel performed the rescue and the response loop closed inside a single day — sensing, decision, and strike compressed into a sequence fast enough that the human deliberative pause, the place where a person reconsiders, had nowhere to sit.CENTCOM^[3]

Three instruments removed, in one week, in climate science, AI safety, and military command. The convergence is the thing worth stopping on. We tell ourselves a story in which the information age makes everything more measurable — more sensors, more dashboards, more telemetry, more oversight. At the edge that actually matters, the opposite is happening. The systems are becoming less observable as they become more consequential, and the loss of sight is not a failure of the instruments. It is a decision to remove them.

I want to give that pattern a name, argue that it is a particular kind of decision problem rather than a measurement problem, and then make the case that the obvious response to it — the one almost everyone reaches for first — is the wrong one, for a reason that complex-systems theory has understood for half a century.

A Different Kind of Not-Knowing

A century ago, Frank Knight drew the distinction that still organizes how we think about not-knowing. Risk is the kind of uncertainty you can measure: you can put a probability on it, price it, insure it. Genuine uncertainty is the kind you cannot, because the future event has no stable distribution to estimate.^[4] Knight's uncertainty is a property of the world. Some things resist measurement in principle, and no amount of effort converts them into risk.

Manufactured unmeasurability is a third thing, and it sits earlier in the sequence than either of Knight's categories. It is not a discovery that some part of the world cannot be measured. It is a choice to stop measuring a part of the world we were already measuring. The AMOC was being tracked. Frontier model behavior was being directly observed. The escalation loop had a human pause in it. In each case, a measurable thing is being moved into the unmeasured column — not by nature, but by a decision. That is why the word is manufactured. We are not merely uncertain. We are building the conditions that prevent us from becoming certain, and we are doing it on purpose, one reasonable step at a time.

The decisive feature is that the choice almost never looks like what it is. Nobody convenes a meeting to vote on becoming blind. Someone lets a planned decommissioning run its course, because halting it midway costs more than finishing it and the current has not collapsed yet. Someone designs a safety architecture that routes danger away from inspection, because that is the cleanest way to contain it. Someone shortens a response loop, because in a contested environment latency is the thing that gets people killed. Each decision is locally correct. The blindness is the sum, and no one decided the sum.

The Decision Underneath the Decision

Here is the structural point that makes this more than a complaint about short-sighted budgeting. Cutting an instrument is not a first-order decision about the world. It is a second-order decision about the conditions under which all of your later first-order decisions will be made. When the agency pulls the AMOC mooring, it is not making a climate decision. It is degrading the evidence base for every climate decision that comes after. When a lab routes risk to an uninspectable model, it is deciding what its own operators will be able to see going forward. When the strike loop compresses the pause, the decision being made is about which decisions will still have a human inside them.

Second-order decisions have a property that makes them treacherous. A first-order decision shows its costs to the person making it; you feel the consequence of a bad call. A second-order decision exports its cost to other people, at other times, who did not make the call. The official who cuts the monitoring line never experiences the blindness — the blindness arrives years later, for whoever is trying to read a current that no longer reports. This is the deep reason nothing in the local process flags the global loss. The loss is not in the local process. It has been shipped downstream.

So what? The most consequential decisions a system makes are often not the visible choices about what to do, but the quiet choices about what it will still be able to see when the time comes to decide.

Three Roads to One Blindness

A skeptic should push back here. Three things happening in one week is a coincidence, not a structure. A climate current, a model router, and a missile loop have nothing to do with each other. The answer is that they do not need to share a cause to share a destination — and the fact that they arrive at the same place by three unrelated roads is exactly what makes the pattern real rather than anecdotal. A coincidence would not have three different generators.

The first road is fiscal. Instruments are pure cost centers with diffuse, deferred benefits, which makes them the easiest target the moment a budget is contested. A mooring produces no revenue and prevents no visible harm this quarter; it merely preserves the ability to see a slow catastrophe coming, and that is a weak hand to play against more urgent claims. The Irminger Sea case shows the mechanism at its sharpest. The monitoring did not lose the budget fight — it won, twice. But the threat of cuts had already set the decommissioning in motion, and a pullout at sea has a lead time that a restored appropriation cannot rewind. The instrument was removed not because the money ran out but because the budget process, once aimed at it, does not reliably stop when the money comes back.

The second road is architectural. Sometimes the move that improves the system and the move that hides it are the same move. Routing a model's most dangerous requests to a separate, guarded system is genuinely safer, and it genuinely reduces what the operators can directly watch. You cannot cleanly separate the safety from the opacity, because they are produced by one decision. The instrument is not cut to save money or time; it is cut because the safest architecture happens to be the least transparent one.

The third road is temporal. In a competition, speed is an advantage, and the deliberative pause — the moment a human being stops to reconsider — is slow by definition. Whatever compresses the loop wins, and the thing the loop compresses is the interval in which measurement and judgment would have happened. The instrument here is not a sensor. It is time. Removing it does not feel like blinding yourself; it feels like getting faster. The blindness is the price of the speed, paid in a currency that does not show up on the invoice.

Fiscal, architectural, temporal: three mechanisms with nothing in common, each terminating in the same place. This is the third instance of the same move in a single week. The convergence is the finding.

The Representational Reflex

Confronted with instruments coming out of the water, almost everyone reaches for the same response: restore them. Fund the moorings. Mandate inspectability. Keep a human in the loop. Rebuild the map. The reflex is so natural that it can be hard to see as a choice at all, but it rests on a specific and contestable theory of what a good decision is — the theory that you decide well by first building an accurate internal model of the world and then choosing the action that the model says is best. Call it the representational view. More information yields a better model, and a better model yields a better decision. On this view, manufactured unmeasurability is simply a catastrophe: it starves the model of the information it runs on.

The representational view is the water we swim in. It underwrites the dashboard, the forecast, the risk model, the situational- awareness display. In stable, decomposable domains it works. The trouble is that it carries a hidden assumption — that a sufficiently complete model is available in principle, so that the only question is whether we are willing to pay for it. For the systems we are actually discussing, that assumption is false, and it has been known to be false for a long time.

There is even a sharp empirical signature of the failure. In work on how artificial agents make decisions under uncertainty, my collaborators and I have found that pouring more information into a decision-maker built on the representational model does not monotonically improve its choices. Past a point, more information makes the choices worse — an inverted-U where the richest map produces the most brittle behavior, because the agent commits ever harder to a representation that the world keeps invalidating.^[5] The pathology is a property of representational decision-making itself, not of information quality. The map gets better and the decisions get worse.

The Map Was Never Going to Be Complete

Why would a more complete representation ever make things worse? Because the systems at issue belong to a class that resists representation in principle, and complex-systems theory has spent decades mapping the reasons. Four of them translate directly to the cases at hand.

The first is that some systems cannot be summarized. The physicist Stephen Wolfram gave this the name computational irreducibility: for certain systems there is no shortcut, no compact formula that tells you where they will end up.^[6] The only way to know what they do is to run them and watch. For a system like that, a predictive dashboard was always a fiction — not a dashboard we failed to build well enough, but a kind of object that cannot exist. The honest instrument for an irreducible system is not a forecast. It is a live feed.

The second is that small differences can grow into large ones. Ever since Edward Lorenz stumbled onto the butterfly effect in a weather model, we have known that some systems are so sensitive to their exact present state that long-range prediction is impossible even when the underlying laws are perfectly known.^[7] The AMOC is the textbook case. A system like this does not give you a gradually reddening gauge before it tips. It gives you a quiet signal and then a steep change, and the only protection is to stay close to the live state — to watch the current continuously, because the warning, when it comes, is brief. This is the watching the mooring provided and the cut mooring removes.

The third is a limit on control itself. The cybernetician Ross Ashby proved what he called the law of requisite variety: to fully regulate a system, a controller must be at least as internally varied as the system it is trying to regulate.^[8] For a genuinely complex system, no controller ever clears that bar. Full control-by-model is not an ambitious target we keep falling short of; it is mathematically out of reach. Which means a strategy that depends on it — decide well by holding a complete-enough model — is not merely expensive. It is aimed at the wrong thing.

The fourth comes from engineering. Control theorists call a system observable if you can infer its internal state from the signals it emits.^[9] Complex systems are only partly observable on their best day; much of their inner state never reaches the surface where an instrument could catch it. This sets the real baseline. We never had complete sight of these systems to begin with. Manufactured unmeasurability is not the loss of a complete picture. It is spending down the partial sight we did have — and partial sight, for a system you can never fully model, is all that keeps you oriented.

For an irreducible system, a predictive dashboard was always a fiction. The honest instrument is not a forecast. It is a live feed.

Put the four together and the representational reflex falls apart on its own terms. The complete map these systems would require does not exist, cannot be computed, would not predict far enough ahead to matter, and would demand a controller more complex than anything we can build. The reason more information made the artificial agent worse is the same reason: it was being used to perfect a representation that the system was never going to hold still for. The lesson is not that measurement is pointless. It is that the point of measurement in a complex system is not to complete a model. It is something else.

Two Ways to Decide

That something else is a second mode of deciding, and naming it is the heart of the matter. If the representational mode tries to build the most complete map it can and then optimize against it, the orienting mode does something different. It treats the goal not as completing a representation but as staying responsive — keeping enough contact with the live system to feel which way it is moving, and keeping enough slack to move with it when it shifts. The orienting decider does not ask “is my model complete?” She asks “am I still in a position to respond when the model fails?”

This is not vagueness dressed up as wisdom. Orienting has a concrete shape, and it differs from representation along specific dimensions. The representational decider invests everything in the accuracy of the picture; the orienting decider invests in three things the picture cannot give her. She maintains a channel for her own confidence in her confidence — a running sense of how much to trust the read, which tightens as signals grow noisy. She keeps a reserve she does not spend, capacity held back precisely so that she can still act when something unmodeled arrives. And she cultivates sensitivity to the dimensions of surprise themselves, tuning her responsiveness rather than her forecast. Where the representationalist seeks to eliminate uncertainty by knowing more, the orienter seeks to remain navigable inside uncertainty that will not be eliminated.

The two modes are two answers to the same second-order question: how should I configure what I will be able to see and do later? Representation answers, build the most complete picture I can afford. Orienting answers, preserve my ability to respond when no picture holds. And this reframes manufactured unmeasurability completely. It is dangerous to both modes, but it is fatal only to the mode that was never going to work on a complex system anyway. The deeper trouble of the present moment is that we are removing instruments under the representational assumption — as if the only loss were a less complete map — while the capacity the orienting mode actually needs is the thing being thrown away.

What Is Actually Being Lost

The orienting frame answers an objection I have been holding off. The three cases are not the same kind of thing. A fifty-year ocean current and a split-second targeting decision do not become “unmeasurable” in the same sense, and gliding over that difference would make this a clever pattern that does not survive contact with the particulars. But seen through the orienting lens, the differences become the point. In each case what is removed is a distinct piece of the same thing — the capacity to stay oriented — and the three pieces are different organs of one faculty.

The AMOC mooring is the minimal continuous signal. Nobody ever had a complete model of the overturning circulation; what the mooring provided was the live feed that let us stay anchored to a current whose warning will be brief. Cutting it does not blur a map we never had. It severs the orientation channel. Fable 5's routing removes a different piece — inspectability. We cannot represent what the model is doing internally, and now we lose the ability to orient to its behavior from the outside as well, opacity stacked on top of a system that was already only partly observable. The strike loop removes the most fundamental organ of all: the deliberative pause is the moment at which re-orientation physically happens, the interval in which a human being notices the map has dissolved and adjusts. Compress it out and the system loses not a sensor but the act of orienting itself.

Three substrates — a signal, an inspection channel, a moment of human judgment — and one structural move: the withdrawal of the capacity to stay oriented. That the substrates differ is not a hole in the argument. It is the evidence that orienting infrastructure is a general thing, with many organs, all of them currently being traded away for cost, for safety, and for speed.

The last of these — the human pause — is worth lingering on, because it is where this connects to a broader thesis about human-AI ensembles. In a well-functioning ensemble, the human and the machine are not redundant; they cover different failure modes, and the human's distinctive contribution is exactly the capacity to notice when the situation has slipped outside the frame the machine is operating in. That noticing takes a moment to occur. Engineer the moment out of the loop in the name of speed, and you have not made a faster version of the same ensemble. You have removed the node that was doing the orienting, and kept only the part that executes against a frame it cannot question.

The Discipline

None of this adds up to a rule that says never remove an instrument. Sometimes you should; monitoring can be genuinely wasteful, some inspection is theater, and a human pause in the wrong place gets people killed. The discipline is not a prohibition. It is a change in the question you ask before you cut.

The representational question is, do we have enough data to model this? For a complex system, that question has no good answer, and leaning on it produces exactly the brittle over-commitment that makes more information make things worse. The orienting question is, are we keeping the capacity to stay oriented when the model fails? Reserve you can deploy when something unmodeled arrives; slack that is not optimized away; a minimal continuous signal anchored to the live state; a preserved human pause where re-orientation can happen — these are not overhead. They are orienting infrastructure, and manufactured unmeasurability is the name for dismantling them one defensible step at a time.

The practical move is to price the future-blindness as a real cost of the present decision. When a choice degrades your later capacity to see — when it pulls a sensor, closes an inspection channel, or compresses out a pause — that degradation belongs on the ledger next to the money saved, the safety gained, or the speed acquired. Right now it sits off the books, because the cost lands downstream on someone who is not in the room. Putting it on the books is the discipline. It will not always change the decision. It will make the decision an actual decision, rather than a blindness that no one chose.

We are removing the instruments under the assumption that the only loss is a less complete map — while the capacity we actually need is the thing being thrown away.

Frank Knight, years before he gave us the modern language of uncertainty, wrote in his 1913 lecture notes that “little or nothing is really fixed but all is a perpetual flux.”^[10] He meant it as a general philosophical claim about causality and substance, not as anything about entrepreneurs — but it doubles as a description of the world the entrepreneur has to act in, a world that will not hold still to be modeled. The systems we have been discussing are that world at its most consequential: a climate current, a frontier intelligence, a theater of conflict, none of them ever fully knowable, all of them now becoming less knowable by choice. Knight's answer was never to wait for certainty that would not come. It was to act well inside the flux. That is the orienting posture, and it has a precondition we are quietly spending down. To stay oriented inside a system you cannot model, you have to keep the few instruments that let you feel where it is going. The most consequential decisions we are making right now are the ones that determine whether, when the time comes, we will be able to feel anything at all.

The instrument that measures the storm is the first thing the storm takes. Lately, we have been handing it over before the storm even arrives.