Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization
Authors: Yurui Pan, Ke Xu, Bo Peng
Summary
The authors propose using topological methods (specifically 0-dimensional persistent homology, which identifies connected components and their lifetimes across scales) to improve LLM alignment. They introduce two techniques: Trajectory Topology Loss (TTL) for supervised fine-tuning, which regularizes the model's update direction to follow "prompt-answer bridges" extracted from the geometry of embeddings, and Topological Preference Optimization (TPO) for DPO, which aligns the improvement direction from rejected to chosen responses with topic-specific semantic vectors. Testing on Qwen2.5-7B-Instruct shows consistent improvements over baselines on preference metrics and LLM-judge evaluations.
Main takeaways:
- Views generation as tracing a trajectory through hidden representation space, not just a sequence of token probabilities
- TTL treats prompt and answer embeddings as a point cloud, uses persistent homology to find topological structure, and aligns model updates with that structure
- TPO constructs semantic preference vectors for topics and guides the DPO update direction in intermediate layers to align with those vectors
- Outperforms non-topological baselines (per-example, nearest-neighbor, random regularizers) on automatic metrics and GPT-judge evaluation
- Maintains or improves toxicity scores while improving preference alignment
Relevance
Directly relevant to my work on installation-path equivalence and behavioral installation methods. This paper explores a novel geometry-aware fine-tuning approach that regularizes how representations move during training, which connects to my experiments on persona geometry collapse (#237) and how different installation methods (fine-tuning, steering, prompts) shape the hidden space differently. The trajectory view could inform thinking about what "persona" means geometrically.
Threat model
Potential threat/caveat for clean result "Stretching turn count, completion length, or system-prompt length at train time fails to amplify marker uptake; the longest system prompt instead leaks across bystander personas (LOW confidence)": this item discusses evaluation.
Abstract
arXiv:2605.07172v1 Announce Type: new Abstract: Alignment of large language models (LLMs) via SFT and RLHF/DPO typically ignores the global geometry of the representation space, relying instead on local token likelihoods or scalar scores. We view generation as tracing a semantic trajectory in hidden space and propose a topology-enhanced alignment framework that regularizes these trajectories using 0-dimensional persistent homology. First, for SFT, we introduce Trajectory Topology Loss (TTL). Treating prompt and gold-answer embeddings as a mixed point cloud, we use a 0D persistent homology algorithm to extract "prompt-answer bridges." TTL aligns the model's actual update direction with these topological bridges rather than arbitrary directions. Second, for DPO, we propose Topological Preference Optimization (TPO). TPO constructs topic-specific semantic preference vectors and aligns the improvement direction between rejected and chosen responses with these vectors in an intermediate hidden layer. We also introduce a dynamic weighting scheme to balance DPO and TPO losses. Evaluating on Qwen2.5-7B-Instruct using UltraChat and Anthropic HH-RLHF, our topology-enhanced objectives consistently outperform strong non-topological baselines (e.g., per-example, nearest-neighbor, random regularizers) on automatic preference metrics and LLM-judge evaluations, while maintaining or improving toxicity. Results show persistent homology and trajectory geometry offer a promising direction for controllable alignment.