GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentations

Authors: Jyotika Singh, Fang Tu, Aziza Mirzadova et al.

arXiv · PDF

Summary

The authors introduce GSM-SEM, a framework for generating math-problem variants with high semantic diversity rather than just surface-level perturbations. Most robustness variants of GSM8K swap names or numbers but keep the underlying facts intact; GSM-SEM modifies entities, attributes, and relationships to produce problems that frequently alter the facts and require recomputation, while constraining generation to preserve the original answer and difficulty. Applying GSM-SEM to GSM8K, GSM-Symbolic, and GSM-Plus, they find consistent performance drops across 14 SOTA LLMs (28% average drop at maximum strictness), with larger declines when semantic perturbations combine with symbolic/plus variations. The framework generates fresh variants on each run, reducing reliance on static benchmarks and lowering memorization bias.

Main takeaways:

• GSM-SEM perturbs problem semantics (entities, attributes, relationships) rather than just surface features, often changing underlying facts while preserving the answer.
• Generates fresh variants stochastically on each run, so it's not a fixed benchmark that models can memorize over time.
• Applied to GSM8K, GSM-Symbolic, and GSM-Plus, producing three fully human-validated SEM datasets.
• 14 SOTA LLMs show consistent performance drops (28% average at maximum strictness), especially when semantic + symbolic/plus perturbations combine.
• Demonstrates applicability beyond math by applying GSM-SEM to BigBenchHard, LogicBench, and NLR-BIRD.

Relevance

Not directly related to my persona or midtraining work. Could be tangentially relevant if I wanted to test whether persona-installed models are more brittle to semantic perturbations than base models—i.e., does fine-tuning a persona make the model more sensitive to problem rewording?

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses bias, robustness, evaluation, benchmark.

Abstract

arXiv:2605.07053v1 Announce Type: new Abstract: Benchmarks like GSM8K are popular measures of mathematical reasoning, but leaderboard gains can overstate true capability due to memorization of fixed test sets. Most robustness variants apply surface-level perturbations (paraphrases, renamings, number swaps, distractors) that largely preserve the underlying facts, and static releases can themselves become memorization targets over time. We introduce GSM-SEM, a reusable and stochastic framework for generating semantically diverse benchmark variants with substantially higher semantic variance than prior approaches. GSM-SEM perturbs problem statements by modifying entities, attributes, and/or relationships, frequently altering underlying facts and requiring models to recompute solutions under new conditions, while constraining generation to preserve the original calculations/answer and approximate problem difficulty. GSM-SEM generates fresh variants on each run without requiring re-annotation, reducing reliance on static public benchmarks for evaluation and thereby lowering the bias of memorization. We apply GSM-SEM on GSM8K and two existing variation suites (GSM-Symbolic and GSM-Plus), producing GSM8K-SEM, GSM-Symbolic-SEM, and GSM-Plus-SEM. Evaluating 14 SOTA LLMs, we observe consistent performance drops with larger decline when semantic perturbations are coupled with symbolic/plus variations (average drop rate 28% in maximum strictness configuration of GSM-SEM). We publicly release the three SEM variants as fully human-validated datasets. Finally, to demonstrate applicability beyond GSM-style math problems, we apply GSM-SEM to additional benchmarks including BigBenchHard, LogicBench, and NLR-BIRD.