NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models
Authors: George Boateng, Naafi Ibrahim, Samuel John et al.
Summary
The authors present NSMQ Riddles, a benchmark of scientific and mathematical riddles from Ghana's National Science and Maths Quiz, a live TV competition for high school students. Each riddle contains at least three clues (with earlier, vaguer clues fetching more points), and answers are numbers, words, or short phrases, enabling automatic evaluation. The dataset spans 11 years and 1,800 riddles covering biology, chemistry, physics, and math. State-of-the-art LLMs (GPT-4o, Gemini 2.0 Pro, Claude Opus 4, and open models like Kimi-K1.5 and DeepSeek-V3) performed worse than the best student contestants, even in high-reasoning settings, showing the dataset is challenging.
Main takeaways:
- 1,800 riddles from an 11-year archive of Ghana's National Science and Maths Quiz, each with 3+ clues of increasing specificity.
- Answers are short (number, word, or phrase), making automatic evaluation straightforward.
- SOTA closed and open LLMs (tested with high and low reasoning settings) perform worse than top student contestants.
- Addresses underrepresentation of Global South datasets in LLM evaluation.
- Contributes a challenging benchmark for scientific and mathematical reasoning beyond Western educational contexts.
Relevance
Not directly related to my persona installation or midtraining work—this is a scientific reasoning benchmark from Ghana. Only tangentially relevant if I ever wanted to test whether persona-installed models maintain reasoning abilities under domain-specific or culturally diverse prompts.
Threat model
Potential threat/caveat for clean result "Stretching turn count, completion length, or system-prompt length at train time fails to amplify marker uptake; the longest system prompt instead leaks across bystander personas (LOW confidence)": this item discusses evaluation, benchmark.
Abstract
arXiv:2605.07051v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown good performance on various science educational benchmarks, demonstrating their potential for use in science and mathematics education. Yet, LLMs tend to be evaluated on science and mathematical educational datasets from the Western world, with an underrepresentation of datasets from the Global South. Furthermore, they tend to have multiple-choice answer options that are trivial to evaluate. In this work, we present NSMQ Riddles, a novel benchmark of Scientific and Mathematical Riddles from Ghana's National Science and Maths Quiz (NSMQ) competition to evaluate LLMs. The NSMQ is an annual live TV competition for senior secondary school students in Ghana that brings together the smartest high school students in Ghana who compete in teams of 2 by answering questions in biology, chemistry, physics, and math over five rounds and five stages until a winning team is crowned for that year. NSMQ Riddles consists of 11 years of riddle questions (n=1.8K) from the 5th round, with each riddle containing a minimum of 3 clues. Students compete to be the first to guess the answer on any of the clues, with earlier clues being vague and also fetching more points. The answers are usually a number, word, or short phrase, allowing for automatic evaluation. We evaluated state-of-the-art models: closed (GPT-5.4, Gemini 3.1 Pro, Claude Opus 4.6) and open models (Kimi-K2.5, DeepSeek-V3.1, GPT-OSS-120B) with high and low reasoning settings. Our evaluation shows that the dataset is challenging even for state-of-the-art LLMs, which performed worse than the best student contestants. This work contributes a novel and challenging benchmark for scientific and mathematical reasoning from the Global South towards enabling a true global benchmarking of LLMs' capabilities for science and mathematics education.