EPS Dashboard

Authors: Arash Shahmansoori

Summary

The authors build a Transformer variant that keeps intermediate representations on a curved geometry (a manifold) using rotation and reflection operations that preserve distances and angles. The key innovation is a way to combine two different mathematical tricks (Cayley transforms for rotation, Householder reflections for flipping) with a learned gate that decides which operation to use, ensuring the network never accidentally distorts its representations. They show this hybrid approach gives better stability during training and handles both types of orthogonal transformations (rotations and reflections) where previous methods could only do one or the other.

Main takeaways:

Orthogonal operations (transformations that preserve vector lengths and angles) help neural networks stay stable during training, especially over long sequences
Standard methods like Deep Delta Learning only guarantee orthogonality at specific settings; this work maintains it for all parameter values by using Cayley transforms
A learned gate decides whether to rotate or reflect representations at each layer, with a penalty encouraging clean either/or choices rather than blending
In experiments with ~1.8M parameters, the architecture showed 1.9× better long-term stability than a competing method and 3.8× better than GPT-style baselines
The method needs 33% fewer layers to achieve similar performance to baselines

Relevance

Not directly related to my persona/behavior installation work — this is about maintaining geometric structure during forward passes in Transformers. Only tangentially relevant if I ever investigate how representation geometry changes during fine-tuning (my result #237 on persona geometry collapse touches this theme).

Abstract

arXiv:2605.06729v1 Announce Type: new Abstract: We present the E$\Delta$-MHC-Geo Transformer, a novel architecture that unifies Manifold-Constrained Hyper-Connections (mHC), Deep Delta Learning (DDL), and the Cayley transform to obtain input-adaptive, unconditionally orthogonal residual connections. Unlike DDL, whose Householder operator is orthogonal only at $\beta \in {0,2}$, our Data-Dependent Cayley rotation $Q(x)=(I+(\beta/2)A(x))^{-1}(I-(\beta/2)A(x))$ preserves orthogonality for all $\beta$ and all inputs. To handle negation, an eigenvalue $-1$ case that Cayley provably excludes, we introduce the E$\Delta$-MHC-Geo Hybrid, which combines Cayley rotation with Householder reflection via a learned operator-selection gate $X'=\gamma(X)Q(X)X+(1-\gamma(X))H_2(X)X$. A midpoint-collapse regularizer, $4\gamma(1-\gamma)$, encourages boundary gate decisions, where each selected component is orthogonal. In matched-parameter comparisons, with approximately 1.79M parameters per model and mean +/- standard deviation over 3 seeds, against four baselines including the concurrent JPmHC, E$\Delta$-MHC-Geo achieves the best long-horizon stability, 1.9x over JPmHC and 3.8x over GPT; the best near-$\pi$ rotation loss, 4.5x over JPmHC on single-plane; strong norm preservation, with 0.001 mean deviation; and 0.96 negation cosine alignment in a diagnostic reflection probe, all with 33% fewer layers. While JPmHC's wider representation excels on pure rotation, its finite Cayley residual mixer excludes an exact $\lambda=-1$ operator and has no reflection branch, motivating our hybrid approach for accessing both connected components of $O(n)$.