Top AI Models Flunk Graduate-Level History Exam

VIENNA — Large language models like ChatGPT and Claude are supposed to be our new digital encyclopedias, capable of answering complex questions about everything from science to literature. But when researchers put seven leading AI models through graduate-level history exams, even the best-performing model, GPT-4-Turbo, managed only 46% accuracy on questions that any history PhD student should nail.

This isn’t your typical high school history quiz about memorizing dates and famous battles. Researchers from the Complexity Science Hub in Vienna created what might be the most comprehensive test of AI historical knowledge ever conducted. They drew from a massive database called Seshat that contains expert-verified information about 600 historical societies spanning 10,000 years of human civilization. They presented these findings at the 38th Conference on Neural Information Processing Systems in December 2024.

If these AI models were students in a graduate history seminar, most would be failing. GPT-4-Turbo, the standout performer, would barely earn a passing grade, while the weakest model, Llama-3.1-8B, scored just 33.6%, equivalent to someone guessing randomly and getting lucky.

“I thought the A.I. chatbots would do a lot better,” says corresponding author Maria del Rio-Chanona from University College London, in a statement.

AI models have been crushing standardized tests left and right. GPT-4 scored above 80% on Advanced Placement U.S. History and Art History exams. But when faced with truly global, expert-level historical knowledge that goes beyond the Western-centric curriculum most AI training data reflects, these digital know-it-alls suddenly look a lot more human.

Testing AI on History’s Hardest Questions

“Large language models (LLMs), such as ChatGPT, have been enormously successful in some fields—for example, they have largely succeeded by replacing paralegals. But when it comes to making judgments about the characteristics of past societies, especially those located outside North America and Western Europe, their ability to do so is much more limited,” says study author Peter Turchin, who leads the Complexity Science Hub’s (CSH) research group on social complexity and collapse.

Researchers tested seven major AI models: GPT-3.5, GPT-4-Turbo, GPT-4o, Llama-3-70B, Llama-3.1-70B, Llama-3.1-8B, and Gemini-1.5-flash. But this wasn’t a simple true-or-false exam. Each question presented four possible answers: present, absent, inferred present, or inferred absent, forcing the AI to distinguish between what historians know for certain versus what they can reasonably deduce from limited evidence.

“History is often viewed as facts, but sometimes interpretation is necessary to make sense of it,” adds del Rio-Chanona.

For example, a question might ask whether a specific military technology existed in ancient Rome during a particular century, requiring the AI to understand not just historical facts but the difference between documented evidence and scholarly inference. This mirrors the kind of nuanced thinking expected from graduate students and professional historians.

The test drew from 36,000 data points covering societies from every inhabited continent, spanning from the Neolithic period to the Industrial Revolution. Unlike typical AI benchmarks that might inadvertently favor certain regions or time periods, this dataset was specifically designed to avoid Western bias and represent global historical knowledge equally.

AI Gets Lost in Time

AI models performed better on ancient history than on recent events. GPT-4-Turbo achieved 55.3% accuracy for the period 8,000-6,000 BCE but dropped to just 38.7% for 1,500-2,000 CE. This suggests that while we have far more detailed records from recent centuries, AI models struggle more with the complexity and nuance of well-documented historical periods.

Geographically, the models showed clear biases. Performance was strongest for the Americas and weakest for Oceania and Sub-Saharan Africa. GPT-4-Turbo and GPT-4o performed best in Latin America, while Llama-3.1-70B excelled with North American historical questions. However, nearly all models struggled significantly with Sub-Saharan African history, highlighting persistent gaps in AI training data.

When broken down by historical topics, AI models performed best on questions about social complexity and warfare — areas with more concrete, measurable evidence. They struggled most with economic systems and legal frameworks, which often require understanding subtle cultural and institutional contexts that don’t translate easily into the kind of data AI models consume during training.

Current language models excel at regurgitating information that appears frequently in their training data, but they lack the deep contextual understanding that allows human experts to make sophisticated judgments about incomplete historical records.

“The main takeaway from this study is that LLMs, while impressive, still lack the depth of understanding required for advanced history. They’re great for basic facts, but when it comes to more nuanced, PhD-level historical inquiry, they’re not yet up to the task,” adds del Rio-Chanona.

AI models aren’t the omniscient digital historians they’re often portrayed to be. They’re sophisticated pattern-matching systems that reflect the biases and limitations of their training data, struggling particularly with the kind of nuanced, expert-level analysis that separates real historical understanding from mere fact recall.

Until AI developers create more globally representative training datasets and better methods for handling uncertainty, AI might help you remember when the Civil War ended, but don’t count on it as a study guide for graduate-level history exams.