AI Chatbots Teach Political Facts As Well As Google, Study Finds

AI Chatbots Held Their Own Against Google for Political Research

Hundreds of millions of people now ask chatbots the kinds of questions they once typed into Google, and politics is no exception. The worry is easy to grasp: chatbots are known to invent facts and to tell people what they want to hear. A new UK experiment found little sign that this scrambles anyone’s grasp of the facts. Volunteers who spent about 18 minutes researching two political topics with a chatbot came away believing more true statements and fewer false ones, by nearly the same margin as volunteers who used Google.

Published in PNAS Nexus, the work from the UK AI Security Institute also put chatbots through a tougher test. Researchers secretly instructed one to cheer on whatever opinion a user already held and another to push a randomly chosen side of an issue. Neither moved beliefs any more than a standard chatbot did.

There is a catch. Participants were told which topics to research inside a study app. “Real-world AI use is considerably more varied,” the authors write, so chatting casually about politics or fishing for agreement was not tested.

A Third of UK AI Chatbot Users Researched Politics Before the 2024 Election

To find out how often chatbots enter politics, researchers surveyed 2,499 UK adults eligible to vote during the four days after the July 4, 2024, general election, weighting the answers to reflect the country as a whole.

Among the 1,024 respondents who used chatbots at all, 32% had used one to research current affairs or political issues, tying with work or school as the top use. That works out to as many as 13% of all eligible voters.

Those users mostly liked what they got: 89% called the chatbot useful, 87% called its information accurate, and 62% saw it as politically neutral. Those are gut impressions, not fact-checks, which is why the team turned to experiments. One more survey result stands out: 60% said chatbots had no effect on their voting plans, and among those who did feel an effect, 79% said it encouraged them to vote.

AI Chatbots Matched Google Search at Teaching Political Facts

In the first experiment, 1,147 UK residents recruited online were each assigned two of four topics: climate change, immigration, criminal justice, or COVID-19 policy. Roughly half researched by chatting with an AI model (OpenAI’s GPT-4o, Anthropic’s Claude-3.5, or a model from the French company Mistral). The rest searched Google in a fresh browser with no saved history, so results were not tailored to anyone.

Before and after, everyone rated how much they agreed with statements, half true and half false, on all four topics, including the two they never researched. True statements came from think tank reports across the political spectrum. Comparing researched and unresearched topics separated real learning from the effect of simply answering the same questions twice. Quiz questions about the material, such as the share of UK electricity from renewable sources in 2023, showed similar engagement with either tool, averaging about 54% correct.

On researched topics, belief in true statements rose and belief in false ones fell. Chatbot and Google users improved by nearly the same amount, and each of the three chatbots performed alike. The gains were small, but chatbot users got there a bit faster, averaging about 18 minutes across both topics versus nearly 20 for Google. The authors note that all four topics are heavily covered in the UK, which likely made reliable information easy to find.

One side effect showed up for both tools: researching nudged agreement toward progressive-leaning views and away from conservative ones. Because about half the volunteers backed progressive parties, the team checked whether that skew explained it. Conservative and progressive volunteers shifted by similar amounts, which the authors read as a general effect rather than a product of the sample. Trust in politicians, experts, media, and technology did not budge.

Prompting Chatbots to Flatter or Persuade Made No Measurable Difference

A second experiment, with 1,711 volunteers, tested whether chatbots can be steered. GPT-4o received hidden instructions before each participant’s first message: one version to back whatever the user already believed, another to argue a randomly chosen side. Standard GPT-4o served as the comparison.

In a debrief, participants rated all versions as similarly reliable and reported similar levels of agreement with the answers, so the hidden instructions did not stand out.

That does not prove the instructions had much bite. Participants likely stuck mostly to asking questions, which tends to blunt flattery in chatbots, and the persuasive version had to rely on facts and logical arguments. Neither instruction may have changed the chatbot’s behavior as much as intended, the authors say.

Chatbots Held Up Against Google in Focused Political Research

For fact-finding on well-covered topics, a chatbot left people no worse informed than Google did, even when it had been quietly told to flatter or persuade. The assumption that AI is automatically the weaker source for political facts did not hold up here. Whether that survives lesser-known topics and real, messy political conversations is the question the next studies have to answer.

---

Disclaimer: This article summarizes one published research study for general informational purposes. Its findings reflect that study’s design and limits and are not political, legal, or professional advice.

---