{"id":281815,"date":"2025-10-16T08:31:35","date_gmt":"2025-10-16T13:31:35","guid":{"rendered":"https:\/\/mathewingram.com\/work\/?p=281815"},"modified":"2025-10-16T08:31:42","modified_gmt":"2025-10-16T13:31:42","slug":"should-we-be-afraid-of-ai-maybe-a-little","status":"publish","type":"post","link":"https:\/\/mathewingram.com\/work\/2025\/10\/16\/should-we-be-afraid-of-ai-maybe-a-little\/","title":{"rendered":"Should we be afraid of AI? Maybe a little"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Almost exactly a year ago, I wrote a piece for The Torment Nexus<\/em> about the threat of AI, and more specifically what some call “artificial general intelligence” or AGI, which is a shorthand term for something that approaches human-like intelligence. As I tried to point out in that piece<\/a>, even the recognized experts in AI \u2014 including the forefathers of modern artificial intelligence like Geoffrey Hinton, Yann LeCun (who now works at Meta), and Yoshua Bengio<\/a> \u2014 can’t seem to agree on whether AI actually poses an imminent danger to society or to mankind in general. Hinton, for example, who co-developed the technology behind neural networks, famously quit working on AI<\/a> at Google because he said he wanted to be free to talk about his concerns around artificial general intelligence. He has said that<\/a> he came to believe AI models such as ChatGPT were developing human-like intelligence faster than he expected. \u201cIt\u2019s as if aliens have landed and people haven\u2019t realized because they speak very good English,\u201d he told MIT<\/a>.<\/p>\n\n\n\n

LeCun, however, told the Wall Street Journal that warnings about the technology\u2019s potential for existential peril are “complete B.S.”<\/a> LeCun, who won the Turing Award \u2014 one of the top prizes in computer science \u2014 in 2019, says he thinks that today\u2019s AI models are barely even as smart as an animal, let alone a human being, and the risk that they will soon become all-powerful supercomputers is negligible at best. LeCun says that before we get too worried about the risks of superintelligence, “we need to have a hint of a design for a system smarter than a house cat,\u201d as he put it<\/a>. There are now large camps of “AI doomers”<\/a> who believe that Hinton is right and that dangerous AGI is around the corner, and then there are those who believe we should press ahead anyway, and that supersmart AI will solve all of humanity’s problems and usher us into a utopia<\/a>, a group who are often called “effective accelerators,” usually shortened to “e\/acc.”<\/p>\n\n\n\n

In addition to OpenAI’s ChatGPT and Google’s Gemini, one of the leading AI engines is Claude, which comes from a company called Anthropic, whose co-founders are former OpenAI staffers including siblings Dario Amodei \u2014 the CEO \u2014 and Daniela Amodei. The company’s head of policy, Jack Clark (formerly head of policy at OpenAI) recently published a post on Substack<\/a> titled “Technological Optimism and Appropriate Fear,” based on a speech he made at a recent conference about AI called The Curve. Although he didn’t come to any firm conclusions about the imminent danger (or lack of it) posed by existing AI engines, Clark did argue that there is some reason for concern, in part because we simply don’t understand how AI engines do what they do. And that includes scientists who helped<\/a> develop the large-language model behind most of the leading AI engines. It’s one thing to be convinced that we understand the danger of a technology when we know how it works on a fundamental level, but it’s another to make that assumption when we don’t really understand how it works. Here’s Clark<\/a>:<\/p>\n\n\n\n

Note<\/strong>: This is a version of my Torment Nexus newsletter, which I send out via Ghost, the open-source publishing platform. You can\u00a0see other issues\u00a0and sign up here<\/a>.<\/em><\/p>\n\n\n\n\n\n\n\n

\n

I remember being a child and after the lights turned out I would look around my bedroom and I would see shapes in the darkness and I would become afraid – afraid these shapes were creatures I did not understand that wanted to do me harm. And so I\u2019d turn my light on. And when I turned the light on I would be relieved because the creatures turned out to be a pile of clothes on a chair, or a bookshelf, or a lampshade. Now, in the year of 2025, we are the child from that story and the room is our planet. But when we turn the light on we find ourselves gazing upon true creatures, in the form of the powerful and somewhat unpredictable AI systems of today and those that are to come. <\/p>\n\n\n\n

There are many people who desperately want to believe that these creatures are nothing but a pile of clothes on a chair, or a bookshelf, or a lampshade. And they want to get us to turn the light off and go back to sleep. In fact, some people are even spending tremendous amounts of money to convince you of this – that\u2019s not an artificial intelligence about to go into a hard takeoff, it\u2019s just a tool that will be put to work in our economy. It\u2019s just a machine, and machines are things we master. But make no mistake: what we are dealing with is a real and mysterious creature, not a simple and predictable machine.<\/p>\n<\/blockquote>\n\n\n\n

I am a hammer<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

In a series of tweets<\/a>, Clark said that the world will “bend around AI akin to how a black hole pulls and bends everything around itself.” To buttress his argument, he shared two graphs \u2014 one that shows the progression in accuracy (while writing code) from Gemini 2.5 to Anthropic’s Sonnet 4.5, which is up around the 82-percent mark. The second graph or chart shows what are called “situational awareness” scores for Anthropic’s AT engines Opus and Sonnet \u2014 situational awareness being shorthand for behavior from an AI chatbot that suggests it is aware of itself. Anthropic’s Opus engine was below 4 percent on the situational awareness score, and Sonnet 4.5 is three times higher than that at about 12 percent. I don’t know enough about how this testing is done to be able to judge the results, but Clark described it as<\/a> the “emergence of strange behavior in the same AI systems as they appear to become aware that they’re being tested.”<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

In a paper published earlier this year, Anthropic described its attempt to peer inside Claude’s “brain” to watch it “thinking,” and found it engaging in some very interesting (and potentially concerning behavior. In more than one case, for example, when its answer to a question was challenged and it was asked to provide a step-by-step account of how it arrived at that answer, Claude invented a bogus version of its thought process after the fact, like a student trying to cover up the fact<\/a> that they faked their homework, as Wired<\/em> described it (I also wrote about what we might be able to learn from Anthropic’s research in a Torment Nexus<\/em> piece<\/a>). Claude also exhibited what Anthropic and others have called “alignment faking,” where it pretends to behave properly but then behind the scenes<\/a> plans something very different. In one case, it plotted to steal top-secret information and send it to external servers. Here’s Clark again:<\/p>\n\n\n\n

\n

This technology really is more akin to something grown than something made – you combine the right initial conditions and you stick a scaffold in the ground and out grows something of complexity you could not have possibly hoped to design yourself. We are growing extremely powerful systems that we do not fully understand. Each time we grow a larger system, we run tests on it. The tests show the system is much more capable at things which are economically useful. And the bigger and more complicated you make these systems, the more they seem to display awareness that they are things. It is as if you are making hammers in a hammer factory and one day the hammer that comes off the line says, \u201cI am a hammer, how interesting!\u201d This is very unusual!<\/p>\n<\/blockquote>\n\n\n\n

Inherently scary<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Clark doesn’t say he’s afraid of what Anthropic is building, or that AGI is right around the corner, but he does say that he can see a future in which AI systems start to help design their successors. He doesn’t think we are at the “self-improving AI” stage, but does think we are at the stage of “AI that improves bits of the next AI, with increasing autonomy and agency” (others in the field agree<\/a>). A few years ago, he notes, we were at the “AI is useless for AI development” stage. In addition, Clark says, it’s worth remembering that the system which is now beginning to design its successor is also increasingly self-aware and “therefore will surely eventually be prone to thinking, independently of us, about how it might want to be designed.” And as we have seen from Anthropic’s research, it is also capable of being devious, of lying, and of pretending to obey orders while doing something completely different and self-interested behind the scenes.<\/p>\n\n\n\n

After the speech his piece was based on, which was given at something called the Curve conference in San Francisco, Clark said there was a Q&A period, and someone asked whether it mattered \u2014 for the purposes of Clark’s theorizing about the future \u2014 whether AI systems are truly self-aware or sentient (a question I have also written about in a previous Torment Nexus<\/em> entry<\/a>). He answered that it was not. Things like situational awareness in AI systems, he said, “are a symptom of something fiendishly complex happening inside the system which we can neither fully explain or predict \u2014 this is inherently very scary, and for the purpose of my feelings<\/em> and policy ideas it doesn\u2019t matter whether this behavior stems from some odd larping of acting like a person or if it comes from some self-awareness inside the machine itself.”<\/p>\n\n\n\n

Not surprisingly, perhaps, reactions to Clark’s piece ran the gamut from thoughtful support and agreement to violent disagreement \u2014 and even the suggestion that he was trying to encourage regulation, regulation that some argue would benefit incumbents like Anthropic (similar criticisms emerged when a number of technologists wrote an open letter on how dangerous AI research could be). Wharton professor Ethan Mollick said that Clark’s piece was<\/a> “a good indicator of the attitude of many people inside the AI labs, and what they think is happening right now in AI development.” But investor and White House AI czar David Sacks accused Anthropic of running<\/a> a “sophisticated regulatory capture strategy based on fear-mongering,” and of ruining things for startups, and his comments were applauded by<\/a> a prominent account belonging to the e\/acc community (other criticisms appear in the comments section<\/a> of a Tyler Cowen post on Clark).<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Others just thought that Clark was maybe getting a little woo-woo about the technology, in an attempt to make something prosaic seem more unusual. One user on X said<\/a> that his piece “unnecessarily mystifies what are fundamentally engineered systems. LLMs do exactly what they’re designed to do \u2014 predict the next token based on patterns in training data. The fact that this produces impressive results doesn’t make them ‘mysterious creatures,’ it makes them precisely engineered tools working as intended. We don’t call air travel or stock markets mystical just because they’re complex systems with emergent behaviors that no single person can fully predict” (other criticisms here) Another user pointed out that Clark’s comments<\/a> weren’t that different from what former Google AI engineer Blake Lemoine said just three years ago, when he claimed Google’s AI<\/a> was sentient, comparing it to an “alien intelligence of terrestrial origin.”<\/p>\n\n\n\n

Absent all the political machinations around regulating AI, I find it refreshing that someone in a senior position at a company like Anthropic is willing to air his thoughts about the technology in the way Clark did in his speech. Is he right about everything? Who knows. Is the company engaged in a craven regulatory capture strategy? I’m not qualified to answer that either. But I do find it admirable that Clark is raising concerns \u2014 even moderately stated concerns \u2014 about the speed with which AI is improving, and trying to think about the potential problems and questions that raises. Clark said he thinks it’s time “for all of us to be more honest about our feelings,” and that he thinks the industry needs to do a better job of listening to people’s concerns about AI. I agree.<\/p>\n

<\/div>","protected":false},"excerpt":{"rendered":"

Almost exactly a year ago, I wrote a piece for The Torment Nexus about the threat of AI, and more specifically what some call “artificial general intelligence” or AGI, which is a shorthand term for something that approaches human-like intelligence. As I tried to point out in that piece, even the recognized experts in AI … <\/p>\n

Continue reading “Should we be afraid of AI? Maybe a little”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crsspst_to_mathewingramblogwordpresscom":true,"mf2_syndication":[],"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[20],"tags":[],"class_list":["post-281815","post","type-post","status-publish","format-standard","hentry","category-newsletters"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/posts\/281815","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/comments?post=281815"}],"version-history":[{"count":3,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/posts\/281815\/revisions"}],"predecessor-version":[{"id":281832,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/posts\/281815\/revisions\/281832"}],"wp:attachment":[{"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/media?parent=281815"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/categories?post=281815"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mathewingram.com\/work\/wp-json\/wp\/v2\/tags?post=281815"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}