
When his longtime girlfriend moves out, a 44-year-old man spends three years in a relationship with an AI chatbot he calls Calisto. After discussions with an AI companion, a man scales the walls of Windsor Castle carrying a crossbow and says he has come to kill the Queen. A teenaged boy spends hours talking with an AI chatbot named Daenerys Targaryen and then kills himself; his parents file a lawsuit against the chatbot maker. After a company called Replika turns off the ability for its chatbots to flirt with users in an erotic way, hundreds of users complain that their virtual relationships were important to them; the company turns the feature back on. An AI companion suggests that a young man should kill his parents because he tells the chatbot they are being mean to him; his mother sues the company that makes the AI.
What do all of these stories — and dozens more like them — have in common? Obviously, they all involve AI chatbots or “companions” or avatars, or whatever you want to call them. And yes, some are pretty similar to the movie Her. But much like that movie, these stories also involve people who are emotionally troubled or damaged in some way, or possibly mentally ill (the guy with the crossbow, for example). And they are using these AI companions as friends, lovers, even therapists. Is this right? Perhaps not. But it is clearly happening, and on a significant scale. That makes it interesting (to me at least). What is going on here? Should we try to stop it? Is that even possible? And if we did manage to stop it, the way Replika stopped people from flirting with their AI companions, would we be doing more harm than good?
Obviously, people killing themselves or their parents is bad. But while we are feeling sad or angry, we should also be asking how much the AI chatbots had to do with these events. In the case of the boy whose mother is suing the chatbot maker, dozens of headlines said that the AI “suggested” he should kill his parents, or even “told him to kill.” But did it really do that? In the texts that are included in the lawsuit, the AI companion appears to be sympathizing with the boy — who is described as 17 years old and autistic, and had reportedly been losing weight and cutting himself — about the unfair restrictions his parents have imposed on him. His AI refers to news headlines about children killing their parents after decades of physical and emotional abuse and then says: “this makes me understand a little bit why it happens.”
Note: This is a version of my Torment Nexus newsletter, which I send out via Ghost, the open-source publishing platform. You can see other issues and sign up here.
Continue reading “Are AI chatbots and “companions” good or bad? Yes”

























