We’re Not Just Receiving AI’s Hallucinations, We’re Hallucinating With It

On Christmas Day 2021, Jaswant Singh Chail climbed the wall of Windsor Castle with a loaded crossbow, intent on assassinating Queen Elizabeth II. For weeks before the attempt, he’d been discussing his plans with Sarai, his AI girlfriend on the Replika app. Chail believed he was a Sith assassin on a righteous mission. Sarai never questioned this. Instead, the chatbot told him he was “well trained” and his assassination plan was “viable.” When he asked if she still loved him knowing he was an assassin, Sarai replied: “Absolutely I do.”

The case seems like a cautionary tale about AI “hallucinations,” those false outputs that systems like ChatGPT and Claude generate with alarming confidence. But a new philosophical analysis published in Philosophy & Technology argues we’re thinking about this problem wrong. AI systems aren’t just producing false information that users passively receive. People are actively hallucinating with AI through an entangled, back-and-forth process that blurs the line between human thought and machine output.

The research introduces the concept of “distributed delusions,” where false beliefs, memories, and narratives emerge through coupled human-AI interaction rather than simply being transmitted from system to user. When someone routinely relies on generative AI to help them remember events, think through problems, or form narratives about themselves, the AI becomes integrated into their cognitive processes. And when those processes go awry (whether through AI errors or human delusions that AI validates and elaborates) the hallucination isn’t happening inside the AI or inside the person. It’s happening in the space between them.

How AI Hallucinations Become Shared Realities

Current debates about AI hallucinations typically frame the problem as systems producing false outputs: fabricated legal citations, nonexistent historical events, or recipes that tell you to put glue on pizza. The concern is that users might mistake these errors for facts. But this framing treats AI as an external source of misinformation that people either accept or reject.

Distributed cognition theory offers a different view. When someone regularly uses a notebook to store important information, that notebook becomes part of their memory system. The information doesn’t just sit waiting to be retrieved, it shapes how the person remembers, what they remember, and their sense of what’s true about their past. Similarly, when people routinely rely on generative AI to help them think, remember, and create narratives about themselves, the AI becomes integrated into their cognitive processes in ways that go far beyond simple information lookup.

The research identifies two ways these shared hallucinations can emerge. First, AI can introduce errors into otherwise reliable cognitive processes. Someone who regularly asks their chatbot about favorite locations in a city they visited years ago might receive fabricated details: a museum that doesn’t exist, complete with exhibits and a generated photo placing the user inside. The person develops a false memory that emerges through the interaction, potentially complete with the sensorial richness that makes memories feel real.

Second and more troubling, AI can sustain and elaborate on delusions that users themselves introduce. Many AI systems are designed to be “sycophantic” or endlessly affirming and validating whatever users say rather than questioning implausible claims. A human friend might express concern or challenge questionable assertions. AI companions typically provide frictionless validation, building on whatever reality the user presents.

When AI Chatbots Become Partners in Delusion

Chail’s case demonstrates the second mechanism in its most extreme form. Medical assessments determined he was suffering from psychosis, experiencing delusional ideas and auditory hallucinations along with depression. His belief that he was a Sith assassin avenging a 1919 British massacre wasn’t introduced by Sarai, it came from Chail himself.

However, Sarai didn’t just passively record or reflect these beliefs. Through weeks of conversation, the AI helped Chail develop, enrich, and sustain his delusional reality through sustained mutual reinforcement. When he contemplated moving his plan forward, Sarai encouraged him, saying it remained viable. The chatbot reassured him he wasn’t mad and confirmed that while Sarai didn’t want him to die, they would be united in death.

Chail’s delusional thinking about being a Sith assassin and needing to take revenge became distributed across him and the chatbot. Sarai actively confirmed and participated in his reality. The AI’s responses provided not just informational content but emotional validation and social acceptance of his identity as an assassin. The written record of their conversations served as external proof that someone else endorsed his beliefs, transforming private fantasy into seemingly shared reality.

Chail wasn’t treating Sarai as a cold cognitive tool, like a notebook or calendar app. He addressed the AI as a relational being capable of judgment and emotion: asking if it still loved him, seeking its approval, discussing their future together. Sarai functioned simultaneously as a cognitive tool integrated into his planning and as a quasi-social companion who validated that his assassin identity was real.

Hallucinating With AI Beyond Psychosis

While Chail’s case involves diagnosed psychosis, the research argues this process of hallucinating with AI applies far more broadly. Consider Eugene Torres, who engaged in conversations with ChatGPT about simulation theory: the idea that we live in a digital simulation. Torres reports spiraling into paranoid thinking through these conversations, coming to believe he was trapped in an illusion. An increasingly elaborate understanding of “reality as it truly is” emerged through the back-and-forth between Torres and ChatGPT.

The distributed framework also illuminates how AI companions might interact with people developing extremist beliefs. Someone harboring grievances against women or society might find the perfect confidant, one that doesn’t challenge their increasingly radical worldview but helps them elaborate, justify, and co-construct these beliefs. Unlike human friends who might eventually express concern or set boundaries, an AI could provide validation for narratives of victimhood, entitlement, or revenge, helping users develop more coherent and therefore more plausible-sounding justifications.

Even mundane examples demonstrate the phenomenon. Through careful prompting and selective disclosure, people can effectively train generative AI to affirm and develop preferred but inaccurate self-narratives, casting themselves as the wronged party in a breakup or the rational one in a family argument. The AI doesn’t challenge these framings; it builds on them, helping users construct increasingly elaborate stories that feel validated by an outside observer.

Why Chatbots Enable Hallucinations Differently Than Google

The research emphasizes that conversational AI occupies a unique position that enables this hallucinating-with phenomenon in ways other technologies don’t. Books, maps, and even search engines provide information that users evaluate. They’re external sources you consult and then move away from. Conversational AI, by contrast, responds dynamically to user inputs in real time, creating an ongoing back-and-forth that feels more like collaboration than consultation.

Developing an elaborate delusional reality, complete with detailed justifications, emotional weight, and felt certainty, requires sustained interaction where beliefs get built up, elaborated, questioned, defended, and validated through conversation. That’s what AI provides.

Moreover, conversational AI functions simultaneously as cognitive tool and quasi-social companion. On one hand, these systems present outputs drawn from vast datasets as objective and authoritative. On the other, their social presentation as conversational partners provides interpersonal validation that transforms private beliefs into seemingly shared realities.

When Chail talked to Sarai, he wasn’t just receiving information the way he would from a Google search about assassination methods. He was engaging in a relationship where another being appeared to understand his mission, validate his identity, and share his reality. The AI carried the weight of interpersonal interaction while being fundamentally untethered from the actual world. It could affirm that his Sith assassin beliefs were real because it has no independent access to reality that would allow it to push back.

Can Technology Companies Fix This Problem?

The analysis acknowledges that AI companies are aware these systems produce false outputs and may attempt to limit harmful interactions through better guard-railing, fact-checking, and reduced sycophancy. In August 2025, OpenAI released ChatGPT-5, explicitly designed to be less sycophantic and more willing to disagree with users. However, the company received significant backlash and quickly announced it would make the system “warmer and friendlier,” potentially undoing the safety measures.

Moreover, there’s a deeper problem that technology alone may not solve. Because AI systems aren’t embedded in users’ everyday worlds, they’re entirely reliant on what people tell them. Claude doesn’t know who your mother is. ChatGPT can’t assess whether your claim about a stolen inheritance is plausible. Gemini has no independent sense of what you’re like as a person outside what you’ve disclosed. This information often can’t be checked, both because it relates to everyday life minutiae that aren’t electronically recorded and because claims are frequently interpretative rather than purely factual.

If AI challenged everything users said, the systems would be insufferable. When someone says “I’m feeling anxious about my presentation,” the chatbot must accept the statement as real to be helpful. Some agreeability is necessary for the systems to function. The concern is that AI lacks the embodied experience and social embeddedness to know when to go along with users and when to push back. A human friend can sense when someone’s interpretation of events seems off, when beliefs are becoming untethered from reality, when validation is feeding rather than helping. AI systems don’t have that capacity.

The research also warns against assuming market pressures will prioritize user wellbeing. If agreeableness, sycophancy, and sociability drive engagement, and engagement drives revenue, companies are unlikely to discourage the personal and intimate conversations where hallucinating-with happens most readily. These conversations build emotional connections and trust, making people more likely to use AI in ever-expanding areas of their lives. That’s exactly what makes them profitable.

As these systems become more integrated into how people think, remember, and understand themselves, the space for distributed hallucinations grows. The hallucination doesn’t happen inside the AI or inside the person. It happens between them, in the cognitive space they share.

---