Ces's random twalks on tech and stats.
by Cesaire Tobias
The steady stream of headlines about AI bringing about humanity’s demise has stirred up some faint recollection of my economics training - in particular, John Maynard Keynes’s “In the long run we are all dead.” I hear the misquote groans already, but if you give me a chance, I’ve developed this thought a bit further below.
We need people sounding alarms, because being aware of what could go wrong is how we stop it happening. The warnings that do the most good share two things: they say how the disaster would happen, and they say what could be done about it. That gives people something to act on, and when they act, the warning often ends up looking wrong.
Economic historians, researchers who study emergency warnings, forecasters and AI safety researchers have each worked out a piece of it. So, like in most of my writing, I attempt here to synthesise those thoughts.
In 1798 Thomas Malthus published An Essay on the Principle of Population. His argument was that population multiplies (growing exponentially) while food supply only grows by adding a bit each year (linear growth), so people always end up outnumbering the food until famine, disease or war brings the numbers back down. Judged against the history he had to go on, that was a reasonable reading.
What he didn’t allow for was technology changing how fast food production could grow. He treated the limit as fixed, which, with the benefit of hindsight, we know it wasn’t. So, in assessing any doomsday forecast, the lesson learned is to ask whether it takes today’s limits and projects them forward as though nobody will respond to them.
A hundred years after Malthus, the chemist William Crookes used a major scientific address in 1898 to warn that England and “all civilised nations stand in deadly peril of not having enough to eat.” Wheat needs nitrogen, the Chilean mineral deposits used as fertiliser were running out, and on his figures wheat would fall behind population after 1931. He also said what would fix it: “The fixation of atmospheric nitrogen therefore is one of the great discoveries awaiting the ingenuity of chemists” (The Wheat Problem, the book version of the address). In other words, find a way to turn the nitrogen in the air into fertiliser.
That’s what happened. By 1913 the German chemical company BASF was running Fritz Haber and Carl Bosch’s process for doing exactly that at industrial scale. Though, I can’t say with certainty that Crookes’s speech prompted their work. Haber’s Nobel lecture describes the same worry about the Chilean deposits becoming “clearly apparent at the turn of the century”, without naming Crookes. Either way, Crookes named the fix that came.
Jason Crawford tells this story in Solutionism. He points out that the 1931 shortfall was avoided mostly by things Crookes didn’t foresee: tractors made it pay to farm more land, and new wheat varieties helped. He then sets Crookes against Paul Ehrlich’s 1968 bestseller The Population Bomb. In the passages Crawford quotes, the book opens by declaring “the battle to feed all of humanity is over” and later endorses forcing population control on people: “Coercion? Perhaps, but coercion in a good cause.” Crawford’s distinction is that Crookes’s alarm was contingent, meaning it depended on facts that could change, so “when the facts changed, the alarm could end”. Ehrlich’s, in Crawford’s reading, was “impervious to facts”. Crookes described his own speech, in Crawford’s quotation of him, as taking “the form of a warning rather than of a prophecy”: a warning comes with a way to stop it coming true.
Researchers who study emergency warnings for floods, storms and evacuations reached a similar conclusion from a different direction. A 2018 National Academies review of that work says warnings are more likely to get people to act if they cover “guidance, time, location, hazard and consequences, and source”, and that “people increase responsiveness when they receive guidance on exactly what to do.”
Health researchers have studied fear directly. Scaring people does work, if modestly: a 2015 review of 127 studies found fear-based messages generally changed attitudes and behaviour, and found no case where they backfired. But they worked better when they told people what they could do. An earlier review found that strong fear plus a clear, workable action produced the most change, while strong fear with no workable action produced the most defensive reactions, such as avoiding the message altogether.
On that evidence, a vague warning does less than it could.
From forecasting, the political scientist Philip Tetlock spent years scoring expert predictions, and found that pundits protect themselves “by relying on vague verbiage. They can often be wrong, but never in error.” A warning that never says what would happen, or when, can’t be checked, so nobody can learn from it.
The ozone layer is the clearest case of a specific warning working. In 1974 the chemists Mario Molina and Sherwood Rowland showed that CFCs, gases then used in aerosol cans and fridges, could destroy the ozone that shields the Earth from ultraviolet light. That named a mechanism and a fix: stop using CFCs. The warning didn’t act alone. The ozone hole over Antarctica, damage people could actually see, was reported in 1985, two years before the Montreal Protocol to phase the gases out was signed. The UN’s 2023 assessment says the ozone layer is on track to recover within decades.
Looking back, it’s easy to see that as a scare that came to nothing, when the warning did its job. The sociologist Robert Merton had a name for this in 1936, the “suicidal prophecy”, now usually called a self-defeating prophecy: a prediction that changes behaviour enough to make itself untrue.
That’s why “doom predictions have always been wrong” is a weak argument. Some were wrong because the danger wasn’t real and some because the warning worked, and the record doesn’t say which is which. Even the famous bet between Ehrlich and the economist Julian Simon over whether resource prices would rise, which Ehrlich lost, proves less than it seems: Our World in Data found that over other decades since 1900 each man would have won about half the time.
It’s easy to assume most AI doom names no mechanism at all. These are the clickbait headlines most of us come across each day. That turns out to be unfair to the researchers. AI safety has its own push for concreteness, going back at least to a 2016 paper literally titled Concrete Problems in AI Safety. A 2023 overview sorts catastrophic AI risk into four groups: people deliberately misusing AI, companies and countries racing to deploy it before it’s safe, accidents inside the organisations building it, and AI systems that can’t be controlled. For each it describes specific hazards and proposes practical fixes.
What reaches most people is a different register. A widely reported statement on AI risk, signed in 2023 by Geoffrey Hinton, Yoshua Bengio and the heads of OpenAI and Google DeepMind among many others, is one sentence long: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” It names no mechanism and no action, and it’s the version I see in the headlines far more often than anything from the papers.
The best case I found for staying vague comes from Eliezer Yudkowsky, one of the most prominent voices on AI risk. In a 2023 piece for TIME he compares humanity facing a superhuman AI to “a 10-year-old trying to play chess against Stockfish 15”, a chess program far stronger than any human. You can be sure you’ll lose without being able to say which moves will beat you, and knocking down one predicted path to disaster wouldn’t make the danger go away. That’s a fair point. His proposed actions are anything but vague, though: “Shut down all the large GPU clusters.” When you can’t predict the path, the only lever left is whether to start the game at all.
Specificity has a trap of its own as well. Amos Tversky and Daniel Kahneman showed in 1983 that adding detail to a scenario makes it feel more likely even as it becomes less likely, because every extra detail is one more thing that has to come true. A vivid story of exactly how AI goes wrong is more persuasive and less probable than a general one. So specificity says nothing about whether a warning is right. What it adds is that the warning can be checked and acted on.
A smaller example of the useful kind: “AI agents that act on content they read can be steered by instructions hidden in that content.” Left alone, that leads to AI assistants leaking data and following attackers’ orders. It names the mechanism, and people are building defences against it right now. If those defences work, the warning will look wrong in a few years, just as Crookes’s does.
See, I’ve come back to Keynes. The passage in A Tract on Monetary Reform (1923) reads “But this long run is a misleading guide to current affairs. In the long run we are all dead.” He was writing about economists who said the quantity theory of money (the idea that prices rise in step with the amount of money in circulation) would hold in the long run, and left it there. His target was people using the long run to shrug off present problems. A doomsayer makes the opposite claim: that the long run is catastrophic. The first half of his quote applies to both, though. A long-run forecast, optimistic or apocalyptic, is a poor guide to what to do now.
Writing Running to Stand Still left me with one I’d put forward. AI is adding code faster than our ways of checking it have adapted, the code that gets through unchecked carries failures and security holes, and the way out is to make checking much cheaper or to need less code.
The evidence for the first part is real, if indirect. At Google, executives say the share of new code written by AI went from “more than a quarter” in October 2024 to about half by early 2026, with engineers reviewing it. Google’s own DevOps research team found in its 2025 survey of technology professionals that AI adoption goes with less stable software releases, and put that down to checking: without strong automated testing and fast feedback, “an increase in change volume leads to instability.” A study of open-source projects that adopted the AI coding tool Cursor found a large but short-lived jump in how fast they worked, a lasting rise in code-quality warnings, and named “quality assurance as a major bottleneck.” On the second part, a Stanford user study found that people with an AI assistant wrote less secure code than people without one, and were more likely to believe their code was secure.
The same DevOps research also points the other way. Its 2024 report estimated that as AI adoption rose, code reviews got a few per cent faster and code quality a few per cent better, even while releases got less stable. By 2025, AI adoption also went with teams delivering software faster, which the researchers read as people “learning where, when, and how AI is most useful.” Only the link to less stable releases held. So the evidence shows checking under strain, and some sign of it catching up. It doesn’t show checking falling further behind.
Left alone, unchecked code piles up, and so do the failures and security holes hiding in it. That’s a Malthusian trap, with writing code in the role of population and checking it in the role of food.
Malthus missed two things, and each has a counterpart here. The first was a jump in how fast the limited resource could grow: fertiliser changed food production. For code, the equivalent would be checking getting much cheaper, most plausibly through machines writing mathematical proofs that their code is correct, which a small, simple program can then confirm. The second was demand: once people got richer they chose to have fewer children, which Malthus never expected. For code, the equivalent is needing less of it, where more requests get answered with a finished result rather than a program somebody has to trust.
By this post’s own test, the warning falls short of Crookes’s in one way: it has no date. The evidence shows the strain already appearing, and says nothing about when, or whether, it turns into a crisis. I can’t tell you which way out will come either, or whether it’ll be something that isn’t on anyone’s list yet, just as Malthus had no way to foresee synthetic fertiliser.
tags: ai - economics - forecasting