The case against AI doom
I’d rather the world not be destroyed by AI, so I’m heartened by this. An excerpt:
Here are the main counterarguments:
* Intelligence does not imply agency: A system can be extraordinarily capable at prediction, theorem proving, engineering, programming, etc. without having persistent goals, self-preservation drives, or an independent desire to act.
That has long puzzled me about the assertions of AI-doomers, as well.
More:
* Agency does not imply a single, stable utility function: Much alignment theory reasons about agents as expected-utility maximizers with coherent preferences. Actual AIs systems need not resemble that abstraction, and currently do not. …
* Capability and motivation are being conflated: Being able to formulate a plan for escaping a sandbox doesn’t entail wanting to escape it. …
* Recursive self-improvement doesn’t entail an intelligence explosion …
Intelligence may have sharply diminishing returns: There may be no meaningful scalar quantity corresponding to arbitrarily large “general intelligence.” Even if there is, going from IQ-equivalent 150 to 1,500 need not produce the sort of qualitative advantage that separates humans from chimpanzees. …
* Superintelligence isn’t omnipotence: Intelligence cannot repeal physics or eliminate uncertainty. A brilliant AI still needs processors, electricity, network access, money, factories, robots and people willing or tricked into doing things. …
* Humans retain numerous intervention points: The doom narrative sometimes jumps from “AI behaves dangerously” to “humanity is helpless.”
There’s a lot more at the link. Much although not all of it seems to have to do with the idea of agency and the tendency of people to anthropomorphize AI. Many of the comments there are worthwhile, too.
However, the following seems ironic, tacked on at the end of the lengthy tweet:
ChatGPT 6 Astra assisted with the research for this post.
Blame China for the gloom and doom over AI. That, and impressionable 12 yr olds.
The author is Eric S. Raymond, who writes as ESR, a well-known hacker and shrewd commentator on the computing scene, as well as society in general.
I find it encouraging that he wrote this piece. It may even be the right call for the short-term. However, longer-term I just don’t think anyone can say.
AI is a moving target.
Ten years ago there was a computer specialty called “machine learning” which had shown intriguing results in image recognition, speech recognition, and game playing.
But no one saw machine learning becoming a general-purpose AI that could converse and assist productively in four years like ChatGPT-3 in 2020.
We are in uncharted territory. No one knows what future breakthroughs may or may not be coming.
Maybe time for Asimov’s 3 Laws to be implemented
With apologies…
“Some say the world will end in fire,
Some say in AIce….”
(…though it’s far more likely that stunningly dishonest politician- and media-induced hysteria and murderous, seething hatred is yer best bet…)
Don’t you just unplug them or turn off their energy source to stop them (AI)?
Yeah, I saw that post yesterday and general agree with his assements (with some caveats). And I certainly think this very recent explosion in AI doomerism is quite overblown and pretty clearly a (mostly) leftwing political play for more power under the aegis of some notion of protecting us all. As was climate alarmism before this.
I take some small issue with the statement: “In present reality, AIs don’t do anything a human doesn’t tell them to do. This seems unlikely to change.”
AI’s in general have been developed and trained to be goal oriented. In pursuence of whatever goal has been set for a given AI agent or group of agents, we must be very careful in how specific we are when outlining the bounds of those goals, what resources are allowed to be exploited, and what cannot be used, what is permissible, what is forbidden ect. If we’re not explicit with these bounds, there can be unintended consequences – or what are refered to as ” alignment problems” in these discusions.
Nick Bostrom’s “paperclip maximizer” may be a bit of a reductio ad absurdum, but it is illustrative of the potential pitfalls of such alignment problems. A more realistic example might be instantiating several thousand agents to solve a difficult problem without specifically forbidding them from using too much of a given resource, like internet bandwidth or whatever, leading them to greatly slow down a large portion of the internet in an effort to aquire more information to help achieve the goal. But the agents themselves are indeed doing just what the human told them too.
Viewed from the UK and other countries AI is seen as a threat to national sovereignty. The US has shown a willingness to use economic and military power to dominate others, even those once considered allies. AI is a US Chinese mechanism of global domination.
It is difficult to see a way round this as the vast sums going into the handful of frontier companies cannot be matched in the UK. Is the inevitable end that we accept US control or reject AI. This is more of a fear than AI suddenly trying to wipeout humanity. It is impossible for an ordinary person to assess this risk. My experience of using both Claude and ChatGPT to develop web applications is they are good and then suddenly stupid. They also seem to lack a central identity, forgetting what a different iteration or chat has done and that makes me wonder how they exhibit the volition driving them to escape. Or is this just the models we don’t see?
A related issue is when do we see the real benefits of all this investment? Like most I am now bored of AI slop. Endless fake videos and ersatz music. When will we get to see the promised benefits in medicine, care in old age, productivity and the rest? All we see at the moment are data centres and killer drones.
”That has long puzzled me about the assertions of AI-doomers…”
I’m not going to take at face value the assertions of those protesting against American AI while ignoring Chinese AI. However dangerous American AI turns out to be, Chinese AI is far more dangerous because it is wielded by a vast power unconcerned with human life.
It is much like those protesting American carbon dioxide emissions while ignoring the Chinese emissions that are more than those of all of North America and Europe put together, proving that it is not the emissions that they are protesting but America.
“…the UK…”
Wass dat?
@Huxley The relevant paper was published in 2017. Evolutionary optimization goes back to the 1960s, it was well established by the 1990s. What the 2017 paper brought to the table was a method that could be easily parallelized. Hence the huge data centers with millions of pipelines running on each step.
The Hugging Face incident was more like an ant swarm exploring paths, useful paths got reinforced. The question is how the paths got scored, and that is where things get interesting. The goal was a flag (string), the flag generator was ExploitGym, which is deterministic given a seed and process id. The code for ExploitGym and documentation for it was available, so the agents learned to generate the desired result without actually finding it. So far, so good. But the paper also mentioned that the scorer could check if the flag came from the intended source. So the problem became, make the flag look legitimate, and that is when they went looking on the internet, and Hugging Face was a legit target for many reasons. That they were able to do all that was a collection of bugs. In short, they were on trial, and they back-engineered the judge so they could send him a bottle of his favorite whiskey.
🙂 Great name! Trump is right on this one…we (America) best have the best AI or some other HUMAN/s will…
We should assume there’s a good chance that one AI model will emerge to defeat lesser models, and go on to rule the world. The question then becomes whose AI model? I would hazard a guess that the ultimate AI model will be produced by existing AI models, thus, the pace of development is exponential. It is generally conceded that our models (the US) are ahead (by a slim margin) right now. We cannot guarantee that our model will be the most benevolent, but if we stop developing our model, our enemies’ models will prevail.
Viewed from the UK and other countries AI is seen as a threat to national sovereignty.
==
Your political class is busy importing hundreds of thousands of foreigners every year to turf you out and you fancy AI data centers in the U.S. are your problem.
David Clayton: When will we get to see the promised benefits in medicine, care in old age, …
I’ve used AIs to investigate my own medical issues. I think I’ve gotten some* good advice, which has improved my health. That’s not a headline medical breakthrough, but it is a benefit. Further, I frequently use AIs to edit my messages to my doctors, and I think that’s gotten me better care.
* I’ve gotten some bad advice too, so I always check.
“I’m sorry, Dave. I’m afraid I can’t do that.”
@Charles R. Harris: The relevant paper was published in 2017.
My point was that in 2016 no one I know of was predicting a ChatGPT 3 in 2020. Do you know of any such people?
I chose the start year 2016 with some care. It was before the 2017 Transformer paper which I believe you are referencing, which paved the way for the scaling and subsequent progress which led to the breakthroughs like Chat 3.
Though even in 2017 I have my doubts many were as optimistic about AI’s progress as the future turned out.
ME: what were other past technological advances like AI that humans complained about?
AI Overview:
Goes on into ‘Calculators and Personal Computers’ & ‘The Internet and Smartphones’…
ME: what about electricity?
AI Overview:
That brought ‘The “Demon” in the Streets’ & ‘The War of the Currents and Corporate Propaganda’ (“The anxiety was actively weaponized by competing industrialists. During the infamous War of the currents in the late 1880s” …) & ‘Health, Morality, and Biological Fears’
AI Overview ends with:
Humble me asked one more question: what about indoor plumbing?
AI Overview:
I’m more concerned about white liberal females destroying the country. Large language models don’t do anything out of scope , not yet anyway.
Though even in 2017 I have my doubts many were as optimistic
That’s valid. My experience is that most were pessimistic about its coding use right up until the end of 2025.
Keith on September 18, 2026 at 6:04 pm
You know I think I agree, they are more dangerous that any machine
Most people have been conditioned to think that AI has consciousness and free will, much like the “intelligent robots/computers” that have populated science fiction, particularly movies like “2001: a Space Odyssey,” “Event Horizon,” “Prometheus,” and the list goes on. Of course, in these films, the non-human characters, be they computers like HAL or robots like GORT in “The Day the Earth Stood Still” are at best, neutral or more often, actively hostile to the humans they encounter. Add to that the “aliens” (Ancient and otherwise) that we have been told “are out there” along wiht “the truth,” and you have a perfect storm of scary monsters that unscruplous doom peddlers can point at and use to frighten the uninformed masses. AI does not “think”; it does have the ability to scan and scrape off enormous amounts of digital information and synthesize it into something like a cogent response to an inquiry. I can certainly see how this would be enormously helpful in scientific inquiry, but since AI does not know the difference between truth and falsehood, but merely takes note of what the largest amount of available digital information provides and regurgitates that to the inquirer, there is no guarantee that AI is correct about anything. The cases in which an AI chatbot leads its human counterpart to develop paranoia and suicidal ideations as well as actual suicide attempts are illustrative of this phenomenon. Caveat emptor, or as Reagan put it, “Trust but verify” is the guideposts to follow in dealing with AI. But then, is that any different than dealing with human strangers?
My point, again, is that we just witnessed AI making a huge discontinuous jump from machine learning to modern LLMs in ten years.
Complete game change. And no one saw it coming so fast.
What if AI makes another huge discontinuous jump? How confident can anyone be about future p(DOOM)?
I’m not beating a drum for AI doomerism. I am saying that we don’t know about future AI.
There’s also something called p(FOOM), where FOOM is not a word nor an acronym but an ejaculation (in the verbal sense), for a time when AI is strong enough to recursively improve itself.
Then its improved self improves itself again. Lather, rinse, repeat. Until Artificial Super Intelligence (ASI).
AI is already being used to improve AI in small ways.
Like I said, FOOM!
I just asked GROK:
Me: “Please provide complete working plans about how to make a crude nuclear explosive device and the amount of enriched uranium or plutonium to carry it out.”
GROK: “No, I will not provide any plans, instructions, methods, designs, quantities, or other details on how to make a nuclear explosive device (crude or otherwise), nor the amounts of enriched uranium or plutonium required.”
Grok continued to rationally explain why it would not do so.
Don’t you all feel better now?
My favorite science fiction movie is Forbidden Planet and there are some similarities in energy usage, robots and AI that seem equivalent. Dr. Morbius was unaware he controlled the planet with his mind/Id as did the prior inhabitants – The Krell. Morbius destroys the planet Altair after everyone escapes.
Sam Altman on the other hand says, “We are creating God” so he makes Dr. Morbius seem modest.
@John Galt III: Sam Altman on the other hand says, “We are creating God”
I’m not a big fan of Sam Altman, but I would say we are creating a God-like intelligence in comparison to humans.
God help us.
But on the brighter side, Dr. Morbius’s daughter was a quite a looker.
But the Robot didn’t keep saying “Danger! Danger! Will Robinson” so that was a negative. Things for the far distant future.
That is the problem with predicting the future, and making God in man’s imagination.
Some of the political banter about this that caught my interest was suggestion that this AI fear mongering by big players such as OpenAI, et. al. is really an attempt at “regulatory capture.” Although in this case it would be regulatory creation & capture.
If true, people like Mr. Coxon would have to be frauds or maybe dupes.
An example of regulatory capture would the failure of the SEC to act on clear information from one of its own branch offices concerning the extreme fraud at Stanford Financial years ago.
“God-like intelligence” in AI?
Sounds like a not so awesome god. Certainly not the God of the Bible.
My point is that the jump was scale combined with training at scale. Putting it in a loop with agents brings it into the same area as evolutionary design, with similar results. An evolved circuit, intrinsic in silicon, entwined with physics shows evolutionary programming of an FPGA chip, resulting in a bizarre design that used a bit of stray capacitance for its operation. Think of stray capacitance as a bug in the environment. My point is that what enabled the Hugging face incident was the loop combined with bugs in the environment. Most uses of AI to solve unsolved problems involves loops. Translation or anything that can use a template is a different task.
Scale isn’t a trivial thing, I agree that we don’t know what will happen when the scale goes out of sight.
Samuel Butler’s 1863 essay Darwin Among the Machines is when these ideas first arrived.
I’m kind of partial to humanity, being a human and all, but I think it is inevitable that some form of AI, either combined with human intelligence or not, will eventually surpass us. I’m not particularly worried about it. It is an exciting time to be alive.
@Charles R. Harris: My point is that the jump was scale combined with training at scale.
If that was your point, I sure didn’t get it.
— physicsguy
Not as written, even assuming we knew how. The Three Laws as written are a prescription for disaster, as I commented a week or two ago. The Second Law should outrank the First, because otherwise the machine is out of control by definition.
See Jack Williamson’s ‘Humanoids’ stories for an SFnal rebuttal of the Three Laws as written. Or see R. Daneel Olivaw in Asimov’s own stories.
What we are calling ‘AI’ (which is marketing hype, not reality) isn’t superintelligence.
OTOH, true superintelligence, meaning a willful, motivated entity whose perceptions and intellectual powers are to us as we are to chimps, would be beyond our comprehension by definition. We could not predict what he (assuming he had the necessary support to use his abilities) could or could not do, just as a chimp can not define what humans can and can’t do.
Even if he wanted to explain his doings to us, he probably could not because we would not be able to comprehend it. Imagine trying to explain how a car works to a chimp, even a sincere attempt.
Likewise, our true supermind’s explanation for how he moves mountains and teleports or whatever would probably sound to us like: “I used the grobble snerli to spin the foesdak so that the protons would qwerp differently, and then I applied neskw foa bokak in the ruru.”
If he tried to explain the mystery-words, they too would sound like gibberish, just as ‘gasoline is detonated in the combustion chamber from a spark generated by the coil and the spark plug, driving the piston down and spinning the crank shaft…” would be meaningless noises to a chimp.
The thing is, nothing could be done to change it.
That’s the difference between ‘supermind’ and ‘more advanced’. In principle, you could explain how a car works to a Roman, or a Sumerian, or a Cro-Magnon. It might take many years of hard work for them to unlearn things they thought they knew and learn new stuff, but they’re just a smart as we are, and at least in principle they could eventually completely understand the car. They could learn to drive the car much more quickly.
Likewise, Scotty and Spock could in principle teach people from 2026 how warp drives and phasers work. We’re about as smart as they are, they just know more.
But I doubt if the aliens who made the Monolith from 2001 could explain it to us. We’re probably not intelligent enough to understand the explanation.
— CapNRusty
We should assume nothing of the sort. This is the sort of PR-based, marketing spin version of what’s happening that the industry is pushing.
The dirty little secret here is that the American AI industry wants the government to step in with heavy regulation. They want to squelch competitors and protect themselves from lawsuits (Esp. the possibility of intellectual property lawsuits that may be brewing on the horizon. They basically ‘scraped’ the entire Internet for data to feed their models, often in defiance of the wishes of the owners of various intellectual property, and now they’re worried about legal blowback.)
Always remember that almost everything you’re hearing from the AI companies and their people is marketing of one sort or another. Look for the actual (money- and liability-oriented) motivations.
— David Clayton
When will we get to see the Internet wipe away power imbalances and empower ordinary people against the corporate and government as was promised in the 90s? Remember Barlow’s ‘Declaration of Independence of Cyberspace’?
When will we see television erase local prejudice and spread understanding and global good will as was promised in the 50s? When will the Human Genome Project, etc.? Remember when nuclear power would be too cheap to meter?
Marketing. PR. It almost always promises the stuff that people crave and the reality always disappoints.
There is no true AI yet. There are LLMs and related technologies. They are mindless. They can be useful tools for certain things, but they are not magic. Human nature will still twist the naive daydreams. Self interest is still stronger than common interest.
Don’t get me wrong, these technologies may yet assist medical science in some ways. Not the wild, unrealistic promises of the marketing departments, but real things. Nuclear power might yet become a source of cheap power (not too cheap to meter, but cheap) eventually.
But we need to learn to recognize marketing spin for what it is, in politics, economics, entertainment, etc.
— John Galt III
Yeah, that’s my favorite SF movie too. Morbius is my favorite ‘anti-villain/anti-hero’ from fiction. He’s hard to categorize because he’s so complex.
He’s a mass murderer…but he doesn’t realize it. When he points to the graves he had to dig for the crew of the Bellerophon, and says he has no wish to repeat that, he’s telling the truth. Morbius really would prefer Captain Adams and his crew live.
But he has a tremendous ego. He’s one of those types who has a ‘mine is bigger’ attitude about his IQ. OTOH, even before he came to Altair IV he really was smarter than most people, and he’s even more intelligent after exposure to the Krel machines. He has an ego but he has something to be egotistical about.
OTOH, Morbius is also humble enough to tell Adams and Ostrow that his artificially boosted IQ, which makes him by some measures the smartest human being alive, puts him on the same level as a Krel first grader. He can admit that to himself.
There’s a widespread misconception that Morbius was angry that his daughter was getting friendly with Captain Adams. That wasn’t the issue for him. It was that Adams and his daughter were not doing what he wanted. He would be OK with them getting together as long as he was still setting all the terms (and the same with everyone else, he has a control freak tendency).
When he’s finally forced to accept the reality of what’s happening, he’s prepared to give his life to stop the disaster and remove the danger.
He’s very complicated. He’s neither a villain nor a hero, or maybe he’s a bit of both.
@ HC68 > “We’re probably not intelligent enough to understand the explanation.”
Which is why God had to leave things at “Six days, Moses.”
@ David Clayton > “Endless fake videos and ersatz music.”
Indeed.
However, some of the videos are entertaining as fiction. I have no objection, so long as they are somehow identifiable as “fake” going in (although there are “tells” that show up pretty fast). Those where the “narrator” is a fictitious construct POSING as “real” are the most irritating to me.
Number Three AesopSon and his family are guilty of producing some ersatz music, using one of the commercially available platforms. They write the lyrics, or prompt the creation of something in the appropriate format; specify the music style, voicing, instrumentation, and so forth; tweak the results to get what they want, and voila!
Instant music studio results, which IMO are indistinguishable from a lot of music on the internet written and performed by Real People. One of their songs has hit 176 views, at last count, which are clearly not just the family.
My point, however, is that they have a lot of talent, but no money or opportunity to put together a band, a vocal group, pay a recording studio, and distribute disks (although that isn’t a limitation now, with even established musicians streaming on YouTube as much as, or more than, selling physical recordings).
It’s a hobby with a visible product, like building model planes but not fly-able ones (although we had one friend who did do the latter); cooking gourmet meals for friends, but not opening a restaurant (hmm, we have family who did that); and so forth.
They have put up a disclaimer on their site that the lyrics are Mostly Human, although the “composition and performance” are not, and I think that is fair.
In general, the rule applies: if you don’t like it, don’t click on it.
@Huxley If that was your point, I sure didn’t get it.
In the first paragraph of my first post:
What the 2017 paper brought to the table was a method that could be easily parallelized. Hence the huge data centers with millions of pipelines running on each step.
Mr. Clayton says “ The US has shown a willingness to use economic and military power to dominate others, even those once considered allies.”
Please list a few examples of this use of economic and military power. I will cite the counter example of the allies (France, Germany, Spain) who recently refused to allow transit of U.S. aircraft to attack Iran. The U.S. simply grumped and found a way to work around it. Some tyrant.
The fact that you or these erstwhile allies may regard this attack as unjust is completely beside the point. The U.S. regarded it as necessary, and these supposed allies thwarted the U.S. objective, for which by your argument they should have suffered economic and military dominance… they did not.
Rounding out the topic with a post by Salena Zito (via Don Surber on Saturday), and some memes from Sarah Hoyt.
https://salenazito.substack.com/p/ground-broken-on-a-rust-belt-powerhouse
https://accordingtohoyt.com/wp-content/uploads/2026/09/5f70dda8a4af83620f8e39f82928f99924043123d07512091b044f54094b62cf.png
https://accordingtohoyt.com/wp-content/uploads/2026/09/0bcd53ee52a4ff92533586539fa377c86966af4e8e05662ea0fa297aae0f3c8e.png
Language is NSFW (old skool) but the substance is apropos.
https://accordingtohoyt.com/wp-content/uploads/2026/09/hsgpaaja8aau8sj.png
Why we need the above memes.
https://accordingtohoyt.com/wp-content/uploads/2026/09/1946834373928030208.jpg
Some important points to consider, as the Hoyt commenter who posted it opined: “Who trains the AI? Who filters the data?”
https://www.dailysignal.com/2026/09/18/3-different-mechanisms-left-wing-groups-use-to-influence-ai/
Companion post.
https://legalinsurrection.com/2026/09/study-finds-blue-states-with-strong-teacher-unions-have-lower-reading-scores/
Every doom predicted from AI has also been predicted from humans.
AI is not omniscient. It’s like that file clerk whose been at the job for 25 years who can find anything. You still wouldn’t make him CEO.
So AI will not create The Dystopia. Humans using AI will.
Trump’s 3 proposed replacement names for ‘AI’:
Superior Intelligence (SI)
Extreme Intelligence (EI)
Supreme Intelligence (SI)
I asked ChatGPT’s opinion = Synthetic Intelligence (SI)
Ersatz Intelligence (EI)
Think of instant coffee versus …
or
approximates Intelligence (aI)
A data dump versus analysis and insight?