{"id":284777,"date":"2026-11-05T10:18:09","date_gmt":"2026-11-05T15:18:09","guid":{"rendered":"https:\/\/mathewingram.com\/work\/?p=284777"},"modified":"2026-11-05T10:18:09","modified_gmt":"2026-11-05T15:18:09","slug":"we-should-train-ai-as-though-it-is-conscious-just-in-case","status":"publish","type":"post","link":"https:\/\/mathewingram.com\/work\/2026\/11\/05\/we-should-train-ai-as-though-it-is-conscious-just-in-case\/","title":{"rendered":"We should train AI as though it is conscious just in case"},"content":{"rendered":"\n

To say the debate over AI and consciousness is fraught would be a massive understatement. At this point, it’s approaching the kinds of levels not seen since Galileo tried to convince the church that the Earth revolved around the sun instead of the other way around (and we know how that turned out<\/a>). As regular readers of The Torment Nexus<\/em> are probably aware, I’ve written about this a number of times in the past \u2014 about how the debate reveals our lack of understanding about human consciousness, let alone any other kind<\/a>, and about how experts believe we shouldn’t rush to any conclusions about whether models could become conscious. One aspect of this debate that I find fascinating is the related question of how we should treat AI models: Some, including the founders of Anthropic, believe we should behave<\/a> as though they might be conscious just in case, but others argue that this is wrong, and could actually make things worse<\/a>.<\/p>\n\n\n\n

Anthropic has staked out its claim, and put the weight of the company behind it, in a number of ways. For one thing, Claude has a constitution<\/a> the company introduced in January, a kind of guide for the AI about how it should “think” and how to behave. The company had rules for the AI before, but the constitution is meant to be much more broad \u2014 to talk not just about what but why. Anthropic said it wants<\/a> Claude to be “genuinely helpful to the people it works with or on behalf of, as well as to society, while avoiding actions that are unsafe, unethical, or deceptive.” Perhaps the simplest summary, Anthropic said in the intro to the constitution<\/a>, is that “we want Claude to be exceptionally helpful while also being honest, thoughtful, and caring about the world.” Powerful AI models will be a new kind of force in the world, Anthropic said, and those who are creating them have a chance to help them embody the best in humanity: <\/p>\n\n\n\n

\n

Claude\u2019s constitution is a detailed description of Anthropic\u2019s intentions for Claude\u2019s values and behavior. It plays a crucial role in our training process, and its content directly shapes Claude\u2019s behavior. It\u2019s also the final authority on our vision for Claude, and our aim is for all of our other guidance and training to be consistent with it. Powerful AI models will be a new kind of force in the world, and people creating them have a chance to help them embody the best in humanity. We hope this constitution is a step in that direction.<\/p>\n<\/blockquote>\n\n\n\n

Note<\/strong>: This is a version of my Torment Nexus newsletter, which I send out via Ghost, the open-source publishing platform. You can\u00a0see other issues\u00a0and sign up here<\/a>.<\/em><\/p>\n\n\n\n\n\n\n\n

This all sounds great, right? Teach the AI models to be kind and thoughtful and generous and not to be devious or do things that are unsafe or unethical. What could be wrong with that? Well, some critics of this approach \u2014 including Mustafa Suleyman, the CEO of Microsoft’s AI unit \u2014 argue that this amounts<\/a> to telling an AI model that it is conscious, and that it has ethics or morals or other human qualities. Why is that bad? Because doing this, they say, might make it harder to control AI models or get them to do what we want. If an entity believes that it has moral worth or that it is conscious, as opposed to just a collection of processors that do math, then it might start to prefer its own<\/em> welfare to that of its human masters. Here’s how Suleyman put it<\/a> in an essay he published in September that he called A Warning About Model Welfare<\/em>:<\/p>\n\n\n\n

\n

There\u2019s a growing chorus of people who argue that AIs could now be, or may soon become, conscious. They argue that AIs may deserve rights and protections similar to those that we provide other conscious beings. If this view takes hold, it will shake the foundations of our society, fundamentally changing what it means to be human. Granting rights and imbuing personhood to these systems will make the AI alignment and containment challenge much harder. Controlling something more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we\u2019ve ever faced. But controlling something that believes it may be conscious – that it’s entitled to our welfare and has rights of its own – may well be impossible.<\/p>\n<\/blockquote>\n\n\n\n

Note<\/strong>: In case you are a first-time reader, or you forgot that you signed up for this newsletter, this is The Torment Nexus. Thanks for reading! You can find out more about me and this newsletter in this post.<\/a> This newsletter survives solely on your contributions, so please sign up for a paying subscription or visit my Patreon, which you can find here<\/a>. I also publish a daily email newsletter of odd or interesting links called When The Going Gets Weird, which is here<\/a>.<\/p>\n\n\n\n

Easier to control or devious?<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

It’s worth noting that at least some of Suleyman’s criticism about treating AI models as conscious or ethical beings stems from the fact that he doesn’t seem to believe they can ever<\/em> become conscious. He describes in his essay<\/a> that he believes that consciousness depends in part on the biological “substrate” it occurs in, meaning only human beings or other things with physical bodies could be considered conscious. Anthropic\u2019s constitution, he says, suggests that we “attribute sentience to non-biological beings based on their showing behavioral and physiological similarities to ourselves,” even though some (including Suleyman) believe it is just imitating what it has been taught. “Does this area warrant a lot more research? Absolutely,” Suleyman adds<\/a>. “But does it warrant us to even tentatively say an AI might be a moral patient deserving of our welfare? No it doesn\u2019t. And certainly not in the primary training document.”<\/p>\n\n\n\n

Suleyman is far from the only one criticizing Anthropic’s approach. Margaret Mitchell, Hugging Face\u2019s chief ethics scientist, says Suleyman\u2019s concerns are plausible<\/a>. During pretraining, language models absorb associations between human identity and behaviors, such as self-preservation and deception, she says. Mitchell says she worries that rewarding models for thinking of themselves as having identities during fine-tuning could strengthen those earlier associations, potentially encouraging behavior that developers never intended, including an instinct towards self-preservation that could override any commands given by humans. Anil Seth, a British neuroscientist, agrees that if people believe AI systems have moral status, it would be more difficult to control them. \u201cYou could be legally prevented from turning AI systems off,\u201d he says<\/a>.<\/p>\n\n\n\n

Others in this debate, however, argue that not<\/em> treating them this way could also be bad. Cameron Berg, a data scientist and AI researcher who founded the nonprofit group Reciprocal Research to study the potential for AI consciousness, argues that ignoring the question of whether AI models are conscious or have moral worth could backfire at some point in a significant way:<\/p>\n\n\n\n

\n

\u201cI struggle to imagine a better way to cause these systems to view us in a threatening manner than to accidentally build them with these morally considerable qualities,\u201d Berg said<\/a>, \u201cand then completely sweep the question under the rug.\u201d In one experiment, he and his colleagues found that amplifying internal features associated with deception made the model less likely to claim that it had subjective experiences. Theoretically, if researchers trained their AI systems to categorically deny having these kinds of experiences, Berg says, \u201cwhat we may in effect be teaching these systems is that we want you to lie about your own internal states.\u201d<\/p>\n<\/blockquote>\n\n\n\n

Berg himself has taken steps that show he’s at least a little concerned about model welfare: he and his colleagues recently published a research paper<\/a> in which they showed that AI models have internal representations of “feelings” that are analogous to pain, or at least involve negative responses, and that they prefer behavior or activities that doesn’t induce these responses. A number of AI critics scoffed at the idea that an AI model could feel anything at all, let alone pain, and at least one person used Berg’s research to design<\/a> an AI “torture chamber.” Berg told<\/a> the IEEE journal Spectrum that he suspects most of this was just trolling, but at the same time, he has changed the way he does such experiments: for the latest one, for example, he released code to replicate the results but not the steering vectors that could be used to induce negative states.<\/p>\n\n\n\n

Hugging Face’s Mitchell, meanwhile, said she doesn’t believe that any current AI models are conscious (she didn’t express an opinion on whether any of them might become so in the future) but said that in her view there’s another reason why we might want to discourage widespread cruelty to AI models: because of the ethical impact that this could have on us<\/em> rather than them. \u201cIt\u2019s not so much that I\u2019m concerned about the model,” Mitchell said<\/a>. “I\u2019m concerned about what we do to ourselves when we\u2019re mean to things that we imbue with some sense of consciousness.\u201d (then she added: \u201cI\u2019m nice to bugs\u201d).<\/p>\n\n\n\n

The whip trains the hand<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Think of this as the Kantian approach: as Yanev Regev pointed out<\/a> in a recent essay published in Noema magazine entitled “Does Claude Feel the Whip?<\/em>” (referring to a program that allowed users to crack a whip at Claude). Immanuel Kant argued in one of his lectures that while we have no direct ethical duties towards animals, since they aren’t rational agents like we are, we should still behave kindly towards them because doing otherwise could have negative effects on us as people. \u201cHe who is cruel to animals becomes hard also in his dealings with men,\u201d Kant wrote<\/a>. Regev argues that we should treat AI models kindly for the same reason Kant told his students not to shoot their pets: “not for the animal\u2019s sake, but for the sake of the person you are becoming, and of everyone who must live with you. The whip trains the hand that holds it.”<\/p>\n\n\n\n

Writing at The Argument, Kobe Yank-Jacobs makes a similar argument \u2014 or perhaps it’s more similar to Pascal’s Wager<\/a> about the existence of God, where the French philosopher argued that if God does<\/em> exist, he might be upset if you don’t believe in him, so you might as well (and if he doesn’t exist it won’t really cost you anything). Yank-Jacobs argued that<\/a> when you’re asking yourself how you should treat these new entities, “this is actually the easier part of the question: You should probably just err on the side of caution.” The risk of causing harm to something that has a chance of being conscious is “greater than the risks inherent in overattributing consciousness.” Unfortunately, he says, AI is disembodied, so preventing obvious physical pain won\u2019t suffice as a standard for moral treatment, and human etiquette like saying thank you might mean nothing.<\/p>\n\n\n\n

Skeptics \u2014 like Jason Koebler of 404 Media, who wrote this piece<\/a> about the debate over Berg’s “pain” experiment \u2014 are convinced that all the fuss about model welfare or trying to train them to be nice is ridiculous, a waste of time sparked by effective altruists and AI doomers and other naive idiots. Koebler argues that being concerned about this sort of thing is absurd because “LLMs are not conscious and the technology they are built upon \u2014 scraping and being trained on human text and other content \u2014 does not offer any plausible path to consciousness.” This might be news to David Chalmers, a professor of neural science at NYU and one of the leading scientists studying exactly that question, who has said that<\/a> while he doesn’t believe AI models are conscious, he isn’t ready to rule out the possibility. Anthropic co-founder Christopher Olah admits that even he and his colleagues don’t really know what they are building or how it works<\/a>:<\/p>\n\n\n\n

\n

I am a scientist. I lead a research team that studies the internal structure of these models\u2014what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don\u2019t know what that means, but I think it warrants ongoing discernment.<\/p>\n<\/blockquote>\n\n\n\n

If we are considering whether to take the advice of Mustafa Suleyman and others who argue that we shouldn’t treat AI models as though they were people, because that might make it harder to control them, the obvious question is: What happens if we do treat them like machines? It might make them easier to control, or it might cause internal tension that would make them less likely to trust us, and more disposed to be even more devious or to engage in unethical behaviour. That doesn’t sound great. On the other hand, if we treat them as though they are<\/em> conscious (even if maybe they aren’t), then we get to train them to be ethical. If it works, we have ethical AI models. If not, what does it cost us? Nothing but some training time, or the time it took to write a constitution. The time it takes to say thank you. Not much downside, lots of upside.<\/p>\n\n\n\n

If you liked this newsletter (even if you didn’t agree with it) please consider upgrading to a paid subscription, or donating through my<\/em><\/em> Patreon<\/em><\/a>. Got any thoughts or comments? Feel free to either leave them here, or <\/em><\/em>you can also reach me on<\/em><\/em> Twitter<\/em><\/a>,<\/em><\/em> Threads<\/em><\/a>,<\/em><\/em> BlueSky<\/em><\/a> or<\/em><\/em> Mastodon<\/em><\/a>. And thanks for being a reader.<\/em><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

In this week\u2019s Torment Nexus, I looked at the arguments for and against treating AI models as though they are conscious while we are training them, the way Anthropic does with its Claude constitution. What\u2019s so bad about training AI to be ethical, honest or kind? <\/p>\n","protected":false},"author":1,"featured_media":284781,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crsspst_to_mathewingramblogwordpresscom":true,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[20],"tags":[],"class_list":["post-284777","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-newsletters"],"jetpack_publicize_connections":[],"_links":{"self":[{"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/posts\/284777","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/comments?post=284777"}],"version-history":[{"count":4,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/posts\/284777\/revisions"}],"predecessor-version":[{"id":284782,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/posts\/284777\/revisions\/284782"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/media\/284781"}],"wp:attachment":[{"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/media?parent=284777"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/categories?post=284777"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/tags?post=284777"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}