SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions
Authors: Tianyu Wang, Nianjun Zhou
Summary
Evaluating literary quality requires assessing interpretive dimensions (cultural representation, emotional depth, philosophical sophistication) that are hard to measure computationally. The authors introduce SAGE, which uses LLMs to evaluate stories across three hierarchical layers (cultural, emotional-psychological, existential-philosophical) with iterative reflection and independent validation. Testing on 100 stories (canonical, pulp fiction, LLM-generated), they achieve 98.8% score convergence and >94% inter-rater agreement. Canonical works consistently outperform pulp and LLM-generated stories on cultural and philosophical dimensions (effect sizes >2.4), but emotional representation shows smaller gaps (d=1.68), suggesting affective patterns are more learnable from training data.
Main takeaways:
- LLM-based evaluation can achieve measurement-grade reliability on complex interpretive dimensions with structured prompts and reflection
- Canonical literature > pulp > LLM-generated on cultural and philosophical depth, with large effect sizes (d>2.4)
- Emotional representation shows smaller gaps (d=1.68), suggesting current LLMs learn affective patterns better than critical stance
- Three quality dimensions (cultural, emotional, philosophical) are empirically distinguishable and moderately correlated (r=0.65-0.68)
Relevance
Not directly related to my persona installation work—this is about evaluating literary quality rather than behavioral conditioning. Included because it's a useful methodology for structured LLM evaluation with high inter-rater reliability, though my focus is on persona behavior rather than content quality.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation.
Abstract
arXiv:2605.07102v1 Announce Type: new Abstract: Evaluating literary quality requires assessing interpretive dimensions such as cultural representation, emotional depth, and philosophical sophistication that resist straightforward computational measurement. We introduce SAGE, a hierarchical evaluation framework that decomposes literary quality into ontology-grounded interpretive dimensions assessed through structured large language model evaluation with multi-round iterative reflection and independent validation. We validate the framework on 100 short stories (50 canonical works, 30 pulp fiction, 20 LLM-generated narratives) across three analytical layers (cultural, emotional-psychological, existential-philosophical) using dual-mode assessment. Across 600 evaluations, the framework achieves 98.8% score convergence and greater than 94% inter-rater agreement, with near-perfect mode invariance between content-based and metadata-based evaluation. Statistical analysis reveals a consistent genre hierarchy (Canonical > Pulp > LLM, all p2.4), while emotional representation shows smaller gaps (d=1.68), suggesting that affective patterns are more learnable from training data than critical stance or philosophical depth. Cross-layer correlations (r=0.649-0.683) confirm the three dimensions capture empirically distinguishable quality facets. These findings demonstrate that theory-driven LLM evaluation can achieve measurement-grade reliability and support systematic identification of where current generative models fall short of human literary production, with direct implications for scalable automated evaluation of open-ended text generation.