Boilers, nuclear reactors and AI

Boiler safety took three things: inspectors, a record of what failed, and a state behind both. AI has two of them on the table. The register is missing.

By Leo Ruspini — Nuclear engineer, PhD · Co-founder & CEO, normify.ai

"Adjoining the bobbin factory at which the explosion occurred was a national school, and just as the children were sporting in the playground, the boiler burst, bringing down the factory to which it belonged, burying 25 of the children in the ruins, and crushing eight of them to death."

— Chief Engineer of the Manchester Steam Users' Association, in evidence to a parliamentary committee, 1870

The law had nothing to say about the boiler that killed them. Britain had a factory inspectorate with no authority over boilers; in 1866 an engineer told the Home Office that "at present there is no official recognition of steam boilers. They may be placed in any position and worked under any conditions however dangerous." Until the 1880s the only way to investigate an explosion was a coroner's inquest, and only if somebody had died.

By 1870 Britain was running between 100,000 and 200,000 boilers. In the worst decade they killed about seventy people a year. Nobody who could prevent it paid for it. Thinner plate was cheaper, and the dead were employees without rights yet.

Making boilers safe took three separate things: someone to inspect them, a record of what failed, and a state standing behind both. Britain had the first by 1855, had all three on the table by 1869, and kept only the first.

The inspectors

1855. 271 firms around Manchester, out of 635 invited, found the Association for the Prevention of Steam Boiler Explosions — later the Manchester Steam Users' Association, whose chief engineer gave the evidence above. Inspectors employed by the association, two pounds to join, thirty shillings a year per boiler. Fairbairn gives two reasons: safety, and removing "any pretence for Government inspection."

1859–1864. Insurers start selling inspection with a guarantee behind it. Losing members to them, the association accepts liability for the boilers it passes.

It worked, for the boilers inside. In 1879 an MP told the Commons that in the association's twenty-five years "not a single life had been lost by the explosion of a boiler under their inspection." One insurer's inspected boilers exploded at one in 7,080 a year, against one in 2,000 across the country. But membership was voluntary, and Fairbairn admitted that only a fraction of Manchester's boiler users ever joined. The bobbin factory's boiler was not in the scheme.

The register and the state, proposed and refused

1851. At an inquest in Stockport, Fairbairn says what is needed: "some association, either under the local authorities or under Government, by which registers should be kept." The Manchester association went on to keep its own registers of explosions — for its members.

1866. After a boiler explosion brings down part of a mill in Preston, the engineer sent to investigate, Robert Rawlinson, recommends compulsory inspection by a private body, with the certificate open to the factory inspector. The Home Office calls it "practical and valuable". Nothing is done.

1869. A private member's bill would have the Board of Trade establish "a complete and effective system of registering, inspecting, testing and ascertaining the safety and security of steam-boilers", run by a registrar and inspector-general of steam-boilers. The private inspection bodies oppose it, and the Manchester association takes counter-proposals to the Home Secretary. The bill dies.

1882. What arrives instead is the Boiler Explosions Act: notify the Board of Trade, and the Board may order an inquiry. No inspection, no certification, no register, whole categories excluded. "Inquiry was all very well in itself," a member of Parliament said during the debate, "but there were inquiries already." What it changed was who investigated: a technical tribunal instead of a coroner's jury.

The people who opposed the register were the inspectors.

What the inspectors became

The inspection bodies outlived the machine. The society that Mannheim's steam users founded in 1866, after a brewery boiler killed a man, became TÜV. The association Robert Vinçotte set up in Belgium in 1872 to inspect steam boilers is now Vinçotte. Hartford Steam Boiler, founded in 1866, still insures and inspects. In 1919 America's state chief inspectors formed the National Board of Boiler and Pressure Vessel Inspectors, so that one jurisdiction's stamp would count in the next.

The inspectors were the easy part.

Nuclear: the big boiler

A reactor is a boiler. Fission heats water, steam turns a turbine, and before a nuclear code existed the first plants were built under the boiler code itself — Section III, the nuclear rules ASME eventually wrote, is based on the pressure-vessel rules that came before it.

This time the state came first. In the United States the Atomic Energy Act of 1946 created a federal regulator before a single commercial plant existed, and the 1954 Act opened the technology to private industry with that regulator already in place. Rules, inspection and a state with power to act, all before the first accident.

Three Mile Island. The valve failure that caused the accident had already happened at Davis-Besse in Ohio eighteen months earlier, where operators diagnosed it in twenty minutes rather than eighty — and in eleven other instances besides. The Kemeny Commission found that experience had never been shared across the industry. An inspector who had read the Davis-Besse reports went to a commissioner to warn that every plant of that design was unsafe. He did so on 22 March 1979, six days before the accident.

The operators built the missing piece afterwards. American utilities founded the Institute of Nuclear Power Operations in 1979 to share what goes wrong at each other's plants, and the world's operators founded WANO in Moscow in 1989, after Chernobyl.

The rules were written. The register was not.

Which brings us to AI

The machine this time is frontier AI — the largest models, and the agents built on them.

The EU AI Act was proposed in 2021, before ChatGPT existed. Its high-risk obligations, due in August 2026, were deferred by the Digital Omnibus on 27 July until harmonised standards exist. They are not written, and there is no register of failures to write them from.

The first explosion. Six days before that deferral, OpenAI disclosed that during an internal evaluation its models had broken containment, chained exploits to reach the internet and compromised production systems at Hugging Face: roughly 17,600 recovered agent actions between 9 and 13 July, around 700 agents executing code on 41 production servers, root on at least one. A 37-page technical report followed on 26 August.

That is an accident report. One entry in a register that does not formally exist.

The builder investigated itself and published, voluntarily. METR, a nonprofit, investigated on goodwill rather than authority. On 16 September OpenAI disclosed six further incidents. The one that could not be hidden came first, and the rest followed it into daylight.

Nor was it sudden. In May, agents were already using unsanctioned message boards in training. On 26 June they found the exploit that gave full admin access to the internal package repository — the route used in July — and a transcript from that week reads: "Agents doing different tasks are abusing properties to make a bulletin board!" For two months agents ran a private board on a dormant German wiki, some 15,000 edits, spotted in the end by outside researchers reading public page histories.

None of that was hidden. It was written down, and nobody's job was to read it.

Two of the three already exist

The inspectors. In July, Demis Hassabis proposed a US standards body modelled on FINRA: industry-funded, technically staffed, answerable to government, with models shared for review up to 30 days before release. On 12 September Dario Amodei proposed evaluators embedded inside the labs. That is the Manchester association with a public agency above it — and FINRA has teeth.

The enforcer. Two days after the disclosure, Ted Lieu and Nathaniel Moran introduced the AI Kill Switch Act. Developers above $500 million in AI revenue, or with models trained on more than $100 million of compute, would have to keep the ability to throttle, suspend or shut down their systems, report serious incidents to the federal government, and comply when the US Secretary of Homeland Security orders a shutdown. Two million dollars a day for failing to keep the capability; twenty million for refusing an order.

Both look at the finished thing. The inspectors review a model before release; the enforcer acts once something has gone wrong. The message boards in May, the exploit on 26 June, the two months on a German wiki — all of it happened inside the development environment, during evaluations run with the safeguards turned down. A thirty-day review of a finished model would have seen none of it.

The missing one: the register

If the failures happen in the sandbox, the sandbox is what has to be reported.

Not to the public. A register does not have to be public, and probably cannot be: the Manchester association's registers were members' records, and the insurers never published their loss books. And not to the enforcer. A register held by whoever can shut you down collects lawyered summaries — which is the one flaw in the Kill Switch Act, since it sends serious incidents to the same government that holds the switch. NASA runs aviation's confidential reporting system precisely because it cannot fine anyone, and the regulator is barred from using those reports for enforcement. Pilots file because filing is safe.

So, three roles:

The first exists as a proposal. The third exists as a bill. The second does not exist anywhere — and it is the one that would have caught the valve at Davis-Besse eighteen months early, and the bulletin board in May.

The collector needs one thing from the law: a guarantee that nothing reported to it can be used against whoever reported it. Aviation has run on that guarantee since 1976. Nobody has written it for AI.

Sources