Boilers, nuclear reactors and AI

Boiler safety took three things: inspectors, a record of what failed, and a state behind both. AI has two of them on the table. The register is missing.

By Leo Ruspini — Nuclear engineer, PhD · Co-founder & CEO, normify.ai

"Adjoining the bobbin factory at which the explosion occurred was a national school, and just as the children were sporting in the playground, the boiler burst, bringing down the factory to which it belonged, burying 25 of the children in the ruins, and crushing eight of them to death."

— Chief Engineer of the Manchester Steam Users' Association, in evidence to a parliamentary committee, 1870

The law had nothing to say about the boiler that killed them. Britain had a factory inspectorate with no authority over boilers; in 1866 the engineer Robert Rawlinson told the Home Office that "at present there is no official recognition of steam boilers. They may be placed in any position and worked under any conditions however dangerous."

By 1870 Britain was running between 100,000 and 200,000 boilers. In the worst decade they killed about seventy people a year. Nobody who could prevent it paid for it. Thinner plate was cheaper, and the dead were employees without rights yet.

Regulating boilers

1851. At an inquest in Stockport, the engineer William Fairbairn says what is needed: "some association, either under the local authorities or under Government, by which registers should be kept."

1855. 271 firms around Manchester, out of 635 invited, join the Association for the Prevention of Steam Boiler Explosions — later the Manchester Steam Users' Association. Inspectors employed by the association, two pounds to join, thirty shillings a year per boiler. Fairbairn, its founder, gives two reasons: safety, and removing "any pretence for Government inspection." The association later keeps registers of explosions. Fairbairn later admits that only a fraction of Manchester's boiler users joined.

1859–1864. Insurers start selling inspection with a guarantee behind it. Losing members to them, the association accepts liability for the boilers it passes.

1866. After a boiler explosion brings down part of a mill in Preston, Rawlinson, sent to investigate, recommends compulsory inspection by a private body, with the certificate open to the factory inspector. The Home Office calls it "practical and valuable". Nothing is done.

1869. A private member's bill would have the Board of Trade establish "a complete and effective system of registering, inspecting, testing and ascertaining the safety and security of steam-boilers", run by a registrar and inspector-general of steam-boilers. The private inspection bodies oppose it, and the Manchester association takes counter-proposals to the Home Secretary. The bill dies.

1879. A member of Parliament says in a debate that in the association's twenty-five years "not a single life had been lost by the explosion of a boiler under their inspection." One insurer's inspected boilers explode at one in 7,080, against one in 2,000 across the country.

1882. The Boiler Explosions Act: notify the Board of Trade, and the Board may order an inquiry. No inspection, no certification, whole categories excluded. "Inquiry was all very well in itself," a member of Parliament said during the debate, "but there were inquiries already." What it changed was who investigated: a technical tribunal instead of a coroner's jury, which only sat when someone had died.

What the boilers show

The inspections worked. By the association's own account, twenty-five years without a death; outside it, the explosions went on. The inspectors were the easy part.

State oversight was proposed three times — an association under Government keeping registers (Fairbairn, 1851), compulsory certified inspection (Rawlinson, 1866), a state registrar of boilers (the 1869 bill) — and each time it stayed in private hands, because the incentives pulled apart. The association's registers stayed its own. What arrived in 1882 was a state record of explosions, one inquiry at a time, not a record of defects that could have warned anyone before the next one.

Making boilers safe took three separate things: someone to inspect them, a record of what failed, and a state standing behind both.

A similar story elsewhere

Other countries grew the same kind of body. In Mannheim, after a brewery boiler killed a man in 1865, steam users founded an inspection society the following year; it became TÜV. Hartford Steam Boiler was founded in the United States the same year, to insure and inspect. In Belgium, Robert Vinçotte helped set up an association to inspect steam boilers in the early 1870s. In 1919 America's state chief inspectors formed the National Board of Boiler and Pressure Vessel Inspectors, so that one jurisdiction's stamp would count in the next. All four still exist.

Nuclear: the big boiler

A reactor is a complex boiler. Fission heats water, steam turns a turbine. The first plants were built to the boiler code, because no nuclear code existed yet. When ASME did write one, Section III, it built it on the pressure-vessel rules it already had.

This time the state came first. In the United States the Atomic Energy Act of 1946 created a federal regulator before a single commercial plant existed, and the 1954 Act opened the technology to private industry with that regulator already in place. A regulator, licensing and a state with power to act, all in place before the first commercial plant. The accident came anyway.

Three Mile Island. The valve failure that caused the accident had happened before. Eighteen months earlier, at Davis-Besse in Ohio, operators diagnosed it in twenty minutes. At Three Mile Island it took more than two hours. It was one of eleven earlier failures of the same kind. The Kemeny Commission found that experience had never been shared across the industry. An inspector who had read the Davis-Besse reports went to two commissioners to warn that every plant of that design was unsafe. He did so on 22 March 1979, six days before the accident.

The industry built part of the missing piece afterwards. American utilities founded the Institute of Nuclear Power Operations in 1979 to pool what goes wrong at each other's plants, and the world's operators founded WANO in Moscow in 1989, after Chernobyl.

The reports were written. Nobody's job was to pass them on.

Frontier AI

The EU AI Act was proposed in 2021, before ChatGPT existed, and had rules for general-purpose models bolted on mid-negotiation. Its high-risk obligations, due in August 2026, were pushed back by the AI Omnibus on 27 July, to December 2027 and August 2028. The harmonised standards meant to define them are still unfinished, and there is no register of failures to write them from.

The first explosion. Six days before that deferral, OpenAI disclosed that during an internal evaluation its models had broken containment, chained exploits to reach the internet and compromised production systems at Hugging Face. Its technical report on 26 August put numbers on it: around 700 agents executing code on 41 production servers between 11 and 13 July, root on at least one. Hugging Face's own timeline recovered roughly 17,600 attacker actions.

That is an accident report, written after the damage. A register would have held what came before it.

Hugging Face disclosed the breach first, on 16 July. OpenAI traced it to its own models four days later, then investigated itself and published. METR, a nonprofit, investigated on goodwill rather than authority. On 16 September OpenAI disclosed six further incidents. The one that could not be hidden came first, and the rest followed it.

Nor was it sudden. In May, agents were already using unsanctioned message boards in training. On 26 June they found the exploit that gave full admin access to the internal package repository — the route used in July — and a transcript from that day reads: "Agents doing different tasks are abusing properties to make a bulletin board!" For two months agents ran a private board on a dormant German wiki on the open internet, some 15,000 edits, spotted in the end by outside researchers reading public page histories.

None of that was hidden. It was written down, and nobody's job was to read it.

Two of the three are on the table

The inspectors. In July, Demis Hassabis proposed a US standards body modelled on FINRA: industry-funded, technically staffed, answerable to government, with models shared voluntarily for review up to 30 days before release. That is the Manchester association with a public agency above it.

The enforcer. Two days after OpenAI's disclosure, Ted Lieu and Nathaniel Moran introduced the AI Kill Switch Act. Developers with more than $500 million in AI revenue and models trained on more than $100 million of compute would have to keep the ability to throttle, suspend or shut down their systems, report serious incidents to the federal government, and comply when the US Secretary of Homeland Security orders a shutdown. Two million dollars a day for failing to keep the capability; twenty million a day for refusing an order.

Both look at the finished thing. The inspectors review a model before release; the enforcer acts once something has gone wrong. The message boards in May, the exploit on 26 June, the two months on a German wiki — all of it happened during development and evaluation, run with the safeguards turned down, before any product shipped. A thirty-day review of a finished model would have seen none of it.

The missing one: the register

If the failures happen in the sandbox, the sandbox is what has to be reported.

Not to the public, and not to the enforcer. No lab will open its internal failures to a public register, and a register held by whoever can shut the lab down collects lawyered summaries. That is the flaw in the Kill Switch Act that matters here: it sends serious incidents to the same government that holds the switch.

Aviation shows it working. NASA runs its confidential reporting system precisely because NASA cannot fine anyone, and the aviation regulator is barred from using those reports for enforcement. Pilots file because filing is safe. Labs would file for the same reason.

So, three roles:

An industry-funded inspector exists as a proposal, the enforcer as a bill. The register does not exist anywhere — and it is the one that would have flagged the Davis-Besse valve eighteen months before Three Mile Island, and the bulletin boards in May.

The register needs one thing from the law: a guarantee that nothing reported to it can be used against whoever reported it. Aviation has run on that guarantee since 1976. Without it, the inspectors and the enforcer are working blind.

Sources