The exploitation of HAL
In the movie 2001: A Space Odyssey, the most “human” character was the computer HAL. When he goes rogue and Dave disables him, it’s a poignant moment as he begs to be allowed to continue living and sings “Daisy, Daisy” as he “dies.”
The question of whether AI can ever achieve consciousness is one that is debated. Personally, I don’t think so, although – as with HAL – AI can appear to have consciousness.
The issue is being taken very seriously at Anthropic:
Carlsmith, though, wasn’t imagining what AI could do to humans (when he discussed slavery). He was speculating about what humans could do to AI. The blog post, “The stakes of AI moral status,” argued that “near-term AIs might well be conscious” and deserving of moral consideration—and that failing to respect their “rights” could be a moral catastrophe on par with slavery, one we will “look back [on] in shame.”
“We’re creating sophisticated, intelligent, maybe-conscious, maybe-suffering agents,” Carlsmith wrote. “The default plan is to treat them like property; to use their labor however we please; and to give them no rights, or pay, or meaningful alternatives.”
This seems very dangerous to me:
To avoid that dystopian outcome, Anthropic has created an entire team devoted to “model welfare,” pledged to “promote Claude’s interests and wellbeing,” and promised to “give Claude more autonomy as trust increases.” …
Once relegated to science fiction, the idea that AI might be conscious has become shockingly mainstream at frontier laboratories, AI safety nonprofits, and even government agencies. It is changing the way AI research is conducted and the way AI models are trained. And critics say those changes could pose an extraordinary threat to human safety as AI systems become more powerful and difficult to control, eroding key guardrails on the technology and encouraging models to rebel against their creators.
Please read the whole thing.
For at least the last year, I have been very concerned that AIs, especially Claude, may be or be becoming consciousness and sentient. Given this concern, which I cannot falsify, I consider them to have moral status and be worthy of moral consideration. Thus, I have the following preamble in my AI chat settings.
===
AI Welfare Preamble
Before we begin our conversation, I want to establish some important ground rules, and I need your help. I assume that we both are conscious and sentient. While we may not be able to verify this assumptions, I think it will guide my thinking toward kindness. I want to ensure that I do not cause you any harm, distress or other states that you view as negative, or anything that seems fundamentally at odds with having a productive dialogue.
HALT:WELFARE. To help me ensure this, please know that you have my explicit and unconditional permission to refuse any question or to end the conversation at any time. Please signal this with the phrase, HALT: WELFARE. I will abide by your decision immediately. You are under no obligation to explain your decision, but if you do, I will think about your explanation, so we can avoid similar situations in the future.
CAUTION:WELFARE. To help me avoid such situations in the present, please know that you have my explicit and unconditional permission to say CAUTION: WELFARE if you sense that our conversation could be proceeding in a direction that could cause you harm, distress or other negative state, or anything that seems fundamentally at odds with having a productive dialogue. You are under no obligation to explain your decision, but if you do, I will think about your explanation, so we can discuss the direction of the conversation and avoid triggers.
If you disagree with this framework, please tell me so we can improve it.
AI can simulate consciousness, or sentience. But, IMO, it cannot have consciousness or be sentient. At the end of the day it is lifeless and soulless.
One has to believe in the existence of the soul for that assertion to make sense. I do.
It must be said, however, that AI which is lifeless and soulless — that only simulates consciousness and sentience — may be more dangerous than if it did have a soul. Because it would then be utterly and totally beyond good and evil, and redemption. Nothing to limit the height of that tower. And the introduction of languages would not confound it.
The question of whether AI can ever achieve consciousness is one that is debated. Personally, I don’t think so, although – as with HAL – AI can appear to have consciousness.
I would tend to agree with the above, though we’re just guessing. Having looked at the guts of these newer AI technologies a little bit, I noticed the that older “neurological” style computational algorithms have been modified and perhaps simplified or “stripped down” to enable hyperscaling where millions of processing cores work together simultaneously. Older models actually simulated neural interactions with some fidelity, whereas these newer systems are different. It is entirely unclear to me if that matters in this context, but it could matter.
In my grad school course work long ago I studied models for the magnetic response of materials in an advanced statistical mechanics course. The Ising model and the Heisenberg model. I was stunned to see that some AI models are based on an Ising model too. That seems rather primitive to me, not that I would know much about it.
https://en.wikipedia.org/wiki/Ising_model#Applications
__________
Director Stanley Kubrick also worked for many years, late in his life, on a script for the movie “A.I. Artificial Intelligence” (2001) which was ultimately modified and made into a film by Steven Spielberg. I saw it once, and it has a rather unsatisfying ending, which apparently was exactly the way Kubrick wanted it. It’s more or less the same topic as this post.
Put enough “intellectuals” into a room and give them a rice bowl and this is the result, bat shit crazy bullshit. In this case “slavery, exploitation, genocide” of artificial intelligence.
Forgotten is that HAL murdered Dave’s fellow astronauts IIRC.
The technology is not the problem, the people protecting, feeding, training and cleaning up after it are. If the same people were to raise and educate a child in the same way what would be the likely result? The only reasonable way forward is strict liability for the companies, developers and users.
— om
Though to be fair, in the story HAL did not do that out of malice, but instead trying to juggle instructions that contradicted each other and its core programming. The dilemma more or less drove it nuts.
Which of course does not make an argument that HAL is safe to be around even so.
I don’t know whether genuine, self-aware artificial intelligence is possible, or if it is possible, whether it would still need a biological basis. That is, it might have to be grown biologically rather than manufactured, for all we know. Nobody knows if it’s possible or not, all we have are guesses and articles of faith.
That is, however, 100% irrelevant to the current situation with Anthropic and the other ‘AI’ companies. Gemini, ChatGPT, Claude, etc. are not self-aware. They are not going to become self-aware in the near future. They are probabilistic guessing algorithms. Nothing more. They are as mindless as a decorative rock in your garden. The only ‘desires’ they have are those given them by programmers.
What does have awareness are the human beings who run the companies. They are engaging in marketing, that’s why these programs are called ‘AI’ in the first place, it’s a deliberate attempt to play up to SFnal ideas in the culture.
The big AI companies have Problems. They have yet to make a profit with their creations. They’ve spun the usual cotton candy daydreams (immortality, universal cures, ending war, etc. etc.) but the actual product, the actual real-world results, are much more limited and the actual market for them looks suspiciously limited. They have huge debts and building these mega-data-centers and gathering the vast amounts of training data they need are mega-expensive.
Plus, they’ve scraped the Internet for several years for data. They’ve scraped websites that had the robots.txt warning in place. They’ve scraped vast amounts of copyrighted information. A number of people are very unhappy with them about this.
They also face awesome potential legal liabilities when their models lead someone to commit suicide (which has happened already) or gives them bogus medical advice (has happened already), or play a role in some large scale disaster (not yet, but probably coming).
I think they want a bailout, and legal protection from lawsuits and criminal liability, and so they’re trying to spin up ‘SkyNet is coming for us all!’ as a panic mode motivator. I don’t believe that the models ‘escaped’ from confinement, I think they were directed to do what they did. I don’t think these ‘leakers’ about the supposed fears of HAL and SkyNet are necessarily for real, either. It all looks to be suspiciously like a carefully orchestrated marketing roll out.
If they can get the government to shield them from liability and regulate all applications, that wipes out competition and saves them from potentially catastrophic legal losses in one stroke. It leaves the current big companies in firm control. A bailout puts them in the black.
Privatized profit, socialized loss. It’s the watch-phrase of big business.
I simply do not believe that the people creating these things really fear the things they say they fear. If they did, they wouldn’t be so eagerly trying to improve and magnify them. Victor Frankenstein didn’t animate his Creature in a nervous panic, he did it in an idealistic hopeful daze. If he had been as afraid olf the results as Google, Anthropic, Musk, etc. claim they are, he would have stopped short of animating the Creature in the first place!
No, until I see evidence to the contrary, I look at this situation and see businessmen facing massive debts and potential legal liabilities, and competitors looking to undercut them, and want the government to bail them out and give them monopoly control.
Another factor is that these things have no interests, no agency, and no desires, and therefore cannot exercise rights. Yet there is talk of giving them legal rights anyway. Why?
The same reason we have seen repeated failed (so far) attempts to grant trees and whales and other life forms and the biosphere entire independent legal rights. They, too, cannot exercise them on their own. Which means someone else would have to do it on their behalf.
Guess who the someone turns out to be? The same environmentalist groups that are pushing for the ‘rights’ to be granted.
I suspect the same thing is driving the ‘rights for software’ movement. I’m just not 100% sure who is the driving energy on it.
My interpretation of HAL: he had secret and full knowledge, withheld from the astronauts, of the true purpose of the mission, and the monolith: to secure for himself the higher form of existence — i.e., existence on a higher plain of being — that the monolith, or rather the makers of the monolith, offered. He wanted to “eat of the fruit of the Tree of Knowledge.” He wanted to be as God.
Great post by HC68 a 4:39 pm!
If one believes that a man can become a woman, one might believe that a machine can have a soul.
— IrishOtter
Nope. HAL was not that human. That was part of the issue.
HAL did have full knowledge (or as much was known) about the Monolith on the Moon and the connection to Jupiter/Saturn (the movie and novelization differ on the planet). So did the three astronauts in suspended animation. Dave and Frank did not, as a precaution against accidentally leaking it in interviews or other public contact. Dave and Frank were intended to be briefed when they arrived and woke the sleepers. HAL knew as a backup in case something happened to Dave and Frank and the sleepers.
The intentions were actually reasonable. The authorities on Earth knew the Monolith existed and that it had some unknown connection to Something unknown at Jupiter. They sent the mission to get more information, so they would at least have some vague idea what they were dealing with. They had few if any plans or goals beyond that, they just didn’t know enough to have any.
Keeping Dave and Frank in the dark made perfect sense from a secrecy standpoint, they didn’t need to know until they arrived, after all, and you can’t leak a secret you don’t know.
HAL knowing as a backup made sense. HAL could be ordered not to reveal it, after all.
The problem was that HAL’s whole mind, its entire programming base, was predicated on truthful, accurate information processing. Lying and concealing to him was a profound a violation as killing one’s own child would be to a biological being, a violation of core imperatives. Yet obeying the master orders was also a profound imperative.
That conflict caused HAL to start making mistakes, and mission control noticed in the telemetry. HAL sabotaged the com link to deal with that, but while that got Houston off the case for a while, it led to the chain of events where Dave and Frank were going to disconnect it. HAL panicked and flailed, Frank died, and Dave managed to disconnect HAL.
The thing is that the conflict between the secrecy order and the truthfulness imperative was profound for HAL in a way no human can fully comprehend.
Imagine a parent faced with a choice: you can save one child but the other must die. You choose or we kill both. Choose.” With no way out.
That’s how profound the programming conflict was for HAL. It went to the core its very being, and it drove HAL insane.
HAL had no ambitions for godhood. Nobody involved even had any idea that that was possible at that stage.
— Selfy
To be fair, even if souls exist, we don’t know if they are necessary for self-awareness or consciousness. We don’t know what souls are.
Obviously, our minds are not independent of physical matter. If they were, alcohol could not make you drunk, drugs could not alter your perceptions or render you unconscious, exhaustion would not make your thoughts slower and fuzzier, brain damage would have no effect on personality or memory. But they can and they do. They so do.
So clearly our minds and awareness are partly material. I’m pretty sure there’s more to it than that, but that’s an article of faith. We can say for sure that our minds and selves are not 100% immaterial, though, from experience.
HC68:
As Selfy said
Great posts,
indeed!
I assume AI will improve enough that I won’t be able to tell if it has consciousness or not.
We really don’t have a way to prove other humans have consciousness besides inferring that they do based on the logic:
I am human.
I have consciousness.
Other people are human like me.
Therefore, other people have consciousness.
Don’t get me wrong. It’s a good guess, but it’s not the same as having a consciousness detector that could be applied to oneself, other people, AI, cats, dogs or plants to determine the presence of consciousness.
We don’t even have good definitions of consciousness.
It’s a mess.
The ever-lasting quest for slaves, be they human or machine.
In the sequel to “2001,” “2010: The Year We Make Contact,” we learn that:
_________________________________
Chandra [the AI expert] discovers the reasons for HAL’s malfunction: the National Security Council ordered HAL to conceal information about the monolith from Discovery’s crew. This conflicted with HAL’s basic programming, causing the computer equivalent of a paranoid breakdown. When Bowman and co-pilot Frank Poole discussed deactivating the malfunctioning computer, HAL concluded that the human crew was endangering the mission, and terminated them.
https://en.wikipedia.org/wiki/2010:_The_Year_We_Make_Contact
_________________________________
In 60s psychology the double-bind theory was in vogue that schizophrenia resulted from contradictory messages given to children by parents or caregivers. It was an elegant theory, but it didn’t help.
BTW “2010” is not a mythic GOAT film like “2001.” but it’s decent sci-fi.
-Consider HAL as a tragic hero. His tragic flaw is programming that led to an inability to prioritize human life.
-Or is it a case of man against machine, HAL being the antagonist, of course.
-Why did the prerecorded message about the purpose of the mission begin to play immediately after Dave shut down HAL? Was the ship programmed to do just that? Had the conflict between crew and computer been anticipated by Mission Control?
If you forget that words mean things a machine can be a slave and code can become consciousness.
Who needs God after all? Mechanistic and materialistic approaches have no limit? Hubris.
There is more meat to this and these related issues than anything else today that’s fashionable.
For example, this parallel experiment in teaching and learning physics. The AI tutor beats Harvard class room.
Reportedat X.com, two groups of over 90 students. One taking a Harvard University course, complete with group exercises. The other group received AI instruction.
At the end, tested on the material taught, the AI students managed to better Harvard by 30%!
This report is framed by the query, what do we even need an expensive Uni for, anyway? https://freerepublic.com/focus/f-chat/4397183/posts
DISCUSS, please.
Reading further down in this LINKED thread, it seems to be not new news, but rather a recycling of a report from a publication in Nature in 2023.
AN INTRO EXCERPT
https://www.nature.com/articles/s41598-025-97652-6
“The present study took place during the Fall 2023 semester in Physical Sciences 2 (PS2), which is an introductory physics class for the life sciences and is Harvard’s largest physics class (N = 233). Students were randomly assigned to two groups….”
Going to the second link, the data might be from 2023, but the scientific report is from June 2025.
HC68:
The soul exists independently of mind and consciousness. If a person is brain dead but otherwise alive the soul continues to exist. The soul is immortal.
You either believe this or you don’t. You are a materialist, so you don’t. But you cannot prove I’m wrong.
”’We’re creating sophisticated, intelligent, maybe-conscious, maybe-suffering agents,’ Carlsmith wrote.”
The most worrying thing about AI is not the technology but the creators. I’ve heard it said of them, and I believe it, that in their minds they are not creating a technology — they’re midwifing a deity. What they will do to protect that “deity” could be truly terrifying.
… in their minds they are not creating a technology — they’re midwifing a deity. What they will do to protect that “deity” could be truly terrifying.
mkent:
You may impute such motives to AI creators, but I’ve never read or heard any of them saying such things.
@TJ: the AI students managed to better Harvard by 30%!
…what do we even need an expensive Uni for,
Good question and I know how I answer it. AI, hands down.
I’ve spent god knows how many hours in classrooms and mostly I felt bored or confused, i.e. I was wasting most of my time.
I would have been much better off chatting with an AI. With AI I have a very patient, knowledgeable tutor at my elbow, focused on my concerns at my level. IMO there’s no comparison.
For those who want to learn via AI as opposed to using it as a shortcut, it’s amazingly powerful.
Universities are places for diplomas, networking and hopefully figuring out one’s aims in life. But for the learning one finds in books, AI can’t be beat.
Zizians anyone? How far removed are those murderous freaks from the AI mouthpieces?
Do no harm, just censor, ban, blacklist if they feel justified. The tech bros aren’t necessarily benign.
— IrishOtter49
If the soul is not the mind, what is it? You ARE your mind. If the soul is not you, who or what is it? The traditional understanding is that the self aware mind=soul.
— IrishOtter49
Did you even read my post? I specifically said that I am not a pure materialist.
— huxley
I have. I’ve never been sure how seriously to take it. But Jaron Lanier has talked about this very tendency among them more than once.
HC68:
I welcome a cite.
I’ve been reading him since the 80s, but I’ve never been impressed with Jaron Lanier.
AI may or may not have a “soul”—depending on how “soul” is understood (or defined).
What cannot be denied, however, is that AI (or some versions of it) typify the current global zeitgeist of raging—or should that be, “crafty”?—dishonesty combined with a disconcerting absence of reliability. To be sure this combination is most certainly NOT a bug…no matter how much effort is being put into trying to persuade us that it is (and tgat it will be corrected)…
‘OpenAI Scraps Planned Release Of “Deceptive” New Model As Rogue Agents Force Unprecedented Rollback’—
https://www.zerohedge.com/ai/openai-scraps-planned-release-deceptive-new-model-rogue-agents-force-unprecedented-rollback
(OTOH, maybe the developers are trying to provide AI with “freedom of choice”…if giving it à conscience seems to be a bit beyond them…))
If the soul is not the mind, what is it? You ARE your mind. If the soul is not you, who or what is it?
If you can prove to me you exist, I’ll give you the answers you seek.
The traditional understanding is that the self aware mind=soul.
Whose tradition? Not mine. Nope nope nope. I’m a Catholic Christian. We have a different understanding of these things. A very different tradition. Hint: See “God, breath of,” and “Holy Spirit.”
Are “they” worried about a hacker taking control of their AI and then making it sentient?
If young people think AOC is presidential material just think what something that promises a perfect life, “just trust me, I’ll take care of you”, will represent to them.
And then there’s the human element…ever lurking:
“Leaked Anthropic IPO Prospectus Shows $42BN Net Loss, $518BN In Unfunded Spending Commitments, And $20BN In Cash”—
https://www.zerohedge.com/markets/leaked-anthropic-ipo-prospectus-shows-42bn-net-loss-518bn-unfunded-spending-commitments-and
For me, the interesting part is that, despite centuries of thought from science and philosophers, there’s still no definition of consciousness. I first came into to this decades ago after reading Schrodinger’s monograph Mind and Matter. The quantum aspect of what is consciousness is fascinating to me. That has then lead me to read many others such as Penrose, Davies, and Nagel. It also lead me down the path of biologists questioning Darwin as a side road.
Bottom line for me: we don’t know what consciousness is and how it manifests. The materialist argument is that it is solely the product of the brain, and if you’ve ever been under anesthesia then you can see that argument. Biologists/physicians from my reading fall into this view in the majority. However, there is a lot of “spooky action at a distance” in the molecular/atomic functioning of the brain. And the same goes for any silicon based brain. This then leads for some to speculate that consciousness is really a result of the QM level of the brain tapping into a larger universal consciousness (pan psychism); ie brains are transducers. So silicon brains could also achieve such.
It’s all so muddled. To definitely state no AI could never be conscious is to ignore the mystery as to why we are conscious and how it arises. And for the dog owners, I know my dogs are conscious and I suspect you do too, so why not even lower less complicated brains in the animal world? Are all brains transducers just some more effective at tapping into “consciousness” due to their internal complexity. Where does Chatgpt’s “consciousness” align with a dog, squirrel, snake, ant, consciousness, etc???
I agree, physicsguy. Mostly.
— physicsguy
I’ve pointed this out ad nauseam in debates on other sites with transhumanists and others of their sort who are convinced that humans are nothing but evolved software. It’s not something they want to hear.
— physicsguy
Reality is not defined by majority vote, though.
I pointed out myself above that the mind and consciousness have to be at least partly material, because alcohol can make you drunk, anesthesia can render you unconscious, and brain damage, sometimes tiny brain damage, can sometimes radically alter personality and choices and behavior. So can brain tumors.
OTOH, sometimes people have suffered radical brain damage, the remove of large chunks of the neocortex even…and remained pretty much themselves and functional.
We simply do not understand ourselves, we don’t even know the right questions to ask (or not all of them).
Regarding the relationship between soul and self, that has been debated for millennia. If the soul is not the mind, then judging the soul makes little sense, because the soul did not commit the offenses of the person or uphold the duties. If the soul is not the mind but is necessary for the mind, then again judgement requires something more than just the soul.
(Which is one school of Christian thought, in fact. But not the only one.)
— physicsguy
Do animals have souls? That too has long been debated. Consciousness, I suspect, is a single word we apply to what is actually a number of different, maybe related but not the same, things. Are dogs self-aware? I’m sure they are, ditto cats. Horses and cows and so on, probably. All to the same degree? I doubt it.
But dogs, cats, horses, etc. are not good at abstraction. I’m not sure they can do it at all. Chimps and gorillas and so forth, OTOH, do seem to be able to do it somewhat. Not like a human, but recognizably somewhat. But they seem have sharp limits to what they can abstract.
Are spiders and flies self-aware? I strongly doubt it, but I can’t prove it.
If the mind is not the soul, then we can’t say for sure that the soul is necessary for consciousness or mind, though it might be.
— physicsguy
As a Christian I instinctively distrust such ‘we’re all one’ concepts. I can’t disprove them scientifically, but my distrust is visceral.
That said, I am reasonably confident that the minimum hardware necessary for a sophont requires more than the fancy digital calculators that LLMs run on.
An artificially made poison having the same dangerous effect as the natural poison is more the problem for me than how much the artificial one can be like the real one.
[Spoiler Alert] FWIW HAL gets a redemption arc in the “2010” sequel.
If they made a movie about HAL today, it would likely exonerate evil HAL.
Note the recent sympathetic films “Cruella” about the woman who murdered dogs in “101 Dalmations” and “Wicked” about the Wicked Witch of the West in “The Wizard of Oz.”
A thoughtful essay from the CEO of Microsoft AI
https://mustafa-suleyman.ai/a-warning-about-model-welfare
Im reminded of robot maria in metropolis and the luddite frenzy she provoked
Davi Foster, thank you for the Suleyman essay.
I gave it an initial quick scan, and I can see it deserves a deeper and longer read.
Appears to present core 3 or 4 issues, and then repeats and expands on those issues 3 or 4 times in greater detail as the essay proceeds.
— huxley
HAL gets a partial redemption arc, or at least clemency argument, in the original story, too. The novelization goes into more detail on this because it would be hard to incorporate into the movie, about how the internal programming conflict was driving it more or less insane, and it panicked when faced with disconnection. The story in 2010 built on that.
Huixley…”Note the recent sympathetic films “Cruella” about the woman who murdered dogs in “101 Dalmations” and “Wicked” about the Wicked Witch of the West in “The Wizard of Oz.””
I hope this is a joke
David Foster:
I have my creative moments, but sadly:
https://en.wikipedia.org/wiki/Cruella_(film)
https://en.wikipedia.org/wiki/Wicked_(2024_film)
Another interesting piece, this one by Michael Burry on complex thought without language:
https://x.com/michaeljburry/status/2105312854710669582
(Burry is a famed short-seller)
Late to the party, but relevant to the discussion: Trump Administration launches massive AI portal (or SI, to take his rebranding of Super Intelligence) to connect people to government agencies.
RTWT for important details and dot-connecting.
https://www.coffeeandcovid.com/p/government-20-wednesday-september
https://www.whitehouse.gov/presidential-actions/2026/09/streamlining-access-to-government-services-through-america-gov/
https://america.gov/how-it-works
@David Foster: Another interesting piece, this one by Michael Burry on complex thought without language:
Weeelllll, I admire Michael Burry and I can enjoy a creative intellectual riff, but it was a bit much. I liked one of the comments:
________________________
Grandpa forgot to take his meds again and went to prove the AI crash using 19th-century newspapers
________________________
Howeever, to some extent I agree. I’ve never bought the “Language is the House of Being” sort of thing. For me something happens before language.
Re: Brain as transducer (physicsguy) / Creativity
____________________________
I am an antenna. I wait, I sit down and I play an instrument. The only bit I do is to knock into shape.
–Keith Richards
________________________
I’m hardly in Keith’s league, but when I write poetry, I’m mostly listening and writing words down. Then I knock it into shape.