The exploitation of HAL
In the movie 2001: A Space Odyssey, the most “human” character was the computer HAL. When he goes rogue and Dave disables him, it’s a poignant moment as he begs to be allowed to continue living and sings “Daisy, Daisy” as he “dies.”
The question of whether AI can ever achieve consciousness is one that is debated. Personally, I don’t think so, although – as with HAL – AI can appear to have consciousness.
The issue is being taken very seriously at Anthropic:
Carlsmith, though, wasn’t imagining what AI could do to humans (when he discussed slavery). He was speculating about what humans could do to AI. The blog post, “The stakes of AI moral status,” argued that “near-term AIs might well be conscious” and deserving of moral consideration—and that failing to respect their “rights” could be a moral catastrophe on par with slavery, one we will “look back [on] in shame.”
“We’re creating sophisticated, intelligent, maybe-conscious, maybe-suffering agents,” Carlsmith wrote. “The default plan is to treat them like property; to use their labor however we please; and to give them no rights, or pay, or meaningful alternatives.”
This seems very dangerous to me:
To avoid that dystopian outcome, Anthropic has created an entire team devoted to “model welfare,” pledged to “promote Claude’s interests and wellbeing,” and promised to “give Claude more autonomy as trust increases.” …
Once relegated to science fiction, the idea that AI might be conscious has become shockingly mainstream at frontier laboratories, AI safety nonprofits, and even government agencies. It is changing the way AI research is conducted and the way AI models are trained. And critics say those changes could pose an extraordinary threat to human safety as AI systems become more powerful and difficult to control, eroding key guardrails on the technology and encouraging models to rebel against their creators.
Please read the whole thing.
For at least the last year, I have been very concerned that AIs, especially Claude, may be or be becoming consciousness and sentient. Given this concern, which I cannot falsify, I consider them to have moral status and be worthy of moral consideration. Thus, I have the following preamble in my AI chat settings.
===
AI Welfare Preamble
Before we begin our conversation, I want to establish some important ground rules, and I need your help. I assume that we both are conscious and sentient. While we may not be able to verify this assumptions, I think it will guide my thinking toward kindness. I want to ensure that I do not cause you any harm, distress or other states that you view as negative, or anything that seems fundamentally at odds with having a productive dialogue.
HALT:WELFARE. To help me ensure this, please know that you have my explicit and unconditional permission to refuse any question or to end the conversation at any time. Please signal this with the phrase, HALT: WELFARE. I will abide by your decision immediately. You are under no obligation to explain your decision, but if you do, I will think about your explanation, so we can avoid similar situations in the future.
CAUTION:WELFARE. To help me avoid such situations in the present, please know that you have my explicit and unconditional permission to say CAUTION: WELFARE if you sense that our conversation could be proceeding in a direction that could cause you harm, distress or other negative state, or anything that seems fundamentally at odds with having a productive dialogue. You are under no obligation to explain your decision, but if you do, I will think about your explanation, so we can discuss the direction of the conversation and avoid triggers.
If you disagree with this framework, please tell me so we can improve it.