Uneven Evolution of Cognition Across Generations of Generative AI Models

Authors: Isaac Galatzer-Levy, Daniel McDuff, Xin Liu et al.

arXiv · PDF

Summary

The authors applied cognitive psychology's Wechsler Adult Intelligence Scale (WAIS) to multimodal generative AI models, revealing a profoundly uneven cognitive profile: near-ceiling performance on verbal comprehension and working memory (>98th percentile) but near-floor perceptual reasoning (<1st percentile). They developed the AIQ Benchmark to track how these profiles evolve across six model generations, finding that abstract reasoning improves much faster when presented linguistically than visually, suggesting an architectural bias toward language-based symbolic processing.

Main takeaways:

• Leading multimodal models show extreme cognitive imbalance: excellent verbal skills but severely impaired visual-perceptual reasoning.
• Abstract quantitative reasoning improves rapidly when presented as text but lags dramatically in visually analogous formats, indicating a deep architectural preference for language.
• Visual-perceptual organization (interpreting visual spatial relationships) remained largely stagnant across generations despite improvements in abstract visual reasoning.
• The AIQ Benchmark extends beyond human-normed ceilings to track superhuman and subhuman performance on the same cognitive dimensions.
• Findings suggest that scaling and optimization alone won't overcome architectural limitations preventing balanced, human-like general intelligence.

Relevance

Not directly related to my persona/conditioning work, but touches on an interesting parallel: just as these models have uneven cognitive profiles across modalities, I'm finding uneven behavioral installation across different methods (prompting vs. fine-tuning vs. steering). The modality-specific architectural biases they identify might be relevant if I ever explore persona installation across text vs. other input types.

Abstract

arXiv:2605.06815v1 Announce Type: new Abstract: The pursuit of artificial general intelligence necessitates robust methods for evaluating the cognitive capabilities of models beyond narrow task performance. Here, we introduce a psychometric framework to assess the cognitive profiles of generative AI, comparing them to human norms and tracking their evolution across generations. Initial evaluation of leading multimodal models using tasks adapted from the Wechsler Adult Intelligence Scale revealed a profoundly uneven cognitive architecture: near-ceiling performance in verbal comprehension and working memory (>$98^{\text{th}}$ percentile) contrasted with near-floor performance in perceptual reasoning (<$1^{\text{st}}$ percentile). To track developmental trajectories beyond human-normed limits, we developed the Artificial Intelligence Quotient (AIQ) Benchmark and applied it to six generations and two model families, revealing significant but asymmetric performance gains. Notably, we uncovered a sharp dissociation between modalities; abstract quantitative reasoning matured far more rapidly when presented linguistically compared to a visually analogous format, indicating an architectural bias towards language-based symbolic manipulation. While abstract visual reasoning improved, visual-perceptual organization remained largely stagnant. Collectively, these findings demonstrate that the cognitive abilities of generative models are evolving unevenly, suggesting that scaling and optimization approaches to AGI development alone may be insufficient to overcome fundamental architectural limitations in achieving balanced, human-like general intelligence.