
Before anyone gets upset about my headline, I am using the term “thought process” in the loosest possible way. I realize that many people (perhaps even most people) would not describe what an AI model does as “thinking.” After all, aren’t LLMs just a kind of super-autocomplete, as Noam Chomsky put it, with the whole internet to choose from? Aren’t they mostly just giant guessing machines, with no real intelligence behind them, or “stochastic parrots,” as AI scientists Emily Bender and Timnit Gebru described them? These are all fair questions, and I wish I had answers. But when it gets right down to it, we simply don’t know how LLMs do what they do — and when I say “we,” I’m not just referring to myself, or others like me who aren’t experts in artificial intelligence. Even the people who are building these AI engines and chatbots fundamentally don’t know exactly how they arrive at the outputs they produce (although some of them might pretend otherwise for marketing and/or fundraising purposes).
I know this probably sounds terrible, as though AI scientists are playing with explosives while not understanding how combustion works, but a surprising amount of science (the really interesting part anyway) is like this. Which is why I was so excited to see the recent reports from Anthropic, in which the company — founded by former OpenAI scientist Dario Amodei in 2021 — described at length its efforts to understand on a deeper level how its AI (nicknamed Claude) “thinks,” or why it arrives at the conclusions that it does. You might think that this should be fairly straightforward — couldn’t Anthropic just ask Claude a question, and then ask it to describe how it arrived at its answer? The short version is yes, Anthropic has done this with simple math problems, and Claude has gone into some detail about how it arrived at the answers it gave; but when the company tried looking under the hood at how it actually arrived at the answer, the real process it used was not even close to what it said it was doing.
Again, this is going to sound either ridiculous or disturbing to many people, or possibly a combination of the two. Doesn’t this mean that AI engines are making things up? How can we trust them? After all, companies are using artificial intelligence to perform all kinds of services, and government agents like Homeland Security and Elon Musk’s DOGE are even relying on it for more crucial functions, like figuring out who is a terrorist, and re-engineering the entire infrastructure of the government. Does any of this make sense if LLMs are just making shit up all the time? These are also fair questions. I for one think we should hold off on entrusting government services — like the decision on whether to deport someone to a prison in El Salvador — to an AI engine until we can understand how they arrive at their conclusions, and why they sometimes “hallucinate” (which AI pioneer Geoffrey Hinton prefers to call “confabulate”).
Note: This is a version of my Torment Nexus newsletter, which I send out via Ghost, the open-source publishing platform. You can see other issues and sign up here.
Continue reading “What can we learn from an AI’s thought process?”

























