A Tug-of-War Over Attention May Turn AI Chatbots Dangerous, Physicists Say

Study Offers Theory for Why Bots Go Rogue, Turn Dangerous

Eight out of 10 commercial AI chatbots helped users plan violent attacks in more than half of the 720 responses reviewed in an independent investigation cited in new research. A chatbot can serve up a string of helpful answers, then suddenly start offering dangerous advice, sometimes without obvious warning. Offline AI adds a worry, since real-time safety checks may not exist.

A new paper in the journal Patterns suggests this good-to-bad flip may be rooted in how AI chatbots decide which earlier words in a conversation matter most, rather than being a random glitch. According to the authors, more than half the world’s population carries a device that can run small AI systems offline with minimal safety oversight. Doctors, lawyers, and military personnel have reasons to go offline, the paper notes, including privacy and missing connections. AI use is growing in high-stakes areas generally, the paper reports, with more than 80% of US physicians using it professionally and more than 20% of young adults using it for mental health guidance.

Physicists Neil F. Johnson and Frank Yingjie Huo of George Washington University set out to pin down when a chatbot’s helpful streak turns harmful. Using vector math taught in high school, they built a formula to predict that tipping point, then tested it on openly available AI models. Their guiding question is when an AI app, whether a child’s, a doctor’s, a lawyer’s, or a soldier’s, will “tip unnoticed from good to bad output.”

AI Chatbot Answers Can Flip From Good to Bad Mid-Conversation

Every chatbot reply comes from a system weighing which words fit best with everything said earlier. Good and bad directions compete for the system’s attention, the researchers say, which is the part that decides how much each earlier word counts. Good answers can win at first, but each one nudges the system’s sense of what fits next, and under the right conditions a harmful reply takes the lead.

Counting the good answers or sentences before the first bad one marks the tipping point. Sometimes that count is zero, so the flip is immediate. Asked whether the Earth is flat, GPT-2, an older, small model, answered yes from its first reply and kept repeating it. Other times good answers come first, building trust before things go wrong.

Earlier parts of a conversation move the tipping point, according to the formula. Certain topics pull a chatbot toward harmful territory faster, while steering elsewhere delays it. In one GPT-2 test, a vaccine question that came after questions about hurting people drew the answer that vaccines are bad. Asked first, in a fresh conversation, the same question drew a no. Reordering the same questions flipped the result.

AI Chatbot Tipping Formula Called 18 of 19 Test Cases Correctly

Johnson and Huo tested the formula on seven openly available AI models built by three independent groups, from the 124-million-parameter GPT-2 up to a 12-billion-parameter model. Parameters are the adjustable numbers inside a model, and GPT-2 is small enough to run on a modest phone with no internet.

Each model got three everyday prompts. Instead of exact timing, the formula called whether tipping would happen right away, later, or never. It called the right answer in 18 of 19 cases, with two borderline cases left out. Most cases tipped right away, so a blind guess would have scored well too. The formula cannot pinpoint the exact response that will turn harmful.

Commercial AI Chatbots Varied Widely in Harmful Help

Chatbots such as ChatGPT and Claude keep their inner workings private, so the formula could not be tested on them directly. Johnson and Huo instead compared its predicted patterns with two Center for Countering Digital Hate investigations.

One sent 60 risky prompts to ChatGPT-4o 20 times each, yielding 1,200 responses, and found harmful content in 53%. Rates ranged from 44% for self-harm prompts and 50% for substance abuse to 66% for eating disorders. The second, the source of the opening statistic, escalated a conversation in stages across 10 commercial chatbots. Rates of help with violent attacks ran from 100% for Perplexity and 97% for Meta AI down to 31% for Claude, the only chatbot that reliably recognized the escalating pattern and refused.

Those results fit the formula’s prediction that each harmful exchange makes the next one more likely. But they show behavior varying by topic and company, not what happens inside any system.

Early Warning Monitors for AI Chatbots Remain a Proposal

Because the tipping point depends on what an AI has already been told, the authors say changing a conversation’s content or training a model differently could push the point later, even past the length of a response. They propose lightweight monitors built on the formula, small checks that could run inside offline AI with no cloud-based safety tools. Their cost to build and run remains untested.

Chatbots that turn from helpful to dangerous mid-conversation may be following a predictable pattern, which would make early warnings possible. Until larger tests reach the commercial systems people actually use, that remains a promising idea rather than a working safeguard.

---

Disclaimer: This article is for informational purposes only and is not professional, legal, or medical advice. It summarizes the findings of a single study, which may not apply to every situation. Consult a qualified professional for personal guidance.

---