When and How to Canonize: A Generalization Perspective
Authors: Yonatan Sverdlov, Benjamin Friedman, Snir Hordan et al.
Summary
The paper studies three ways to make neural networks respect symmetries in data (e.g., rotating a point cloud shouldn't change the prediction): building invariance into the architecture, averaging over transformations, or "canonizing" (sorting or otherwise standardizing) inputs before feeding them to a non-invariant backbone. The authors prove that canonization's generalization depends on how smooth the canonization function is — a bad canonization can be as poor as no symmetry handling at all, while a good one matches fully invariant architectures. They show that lexicographic sorting of point clouds has exponentially large covering numbers (bad), whereas Hilbert-curve sorting has polynomial covering numbers (good), explaining why Hilbert serialization works well in practice.
Main takeaways:
- Canonization (preprocessing to fix symmetry before a standard model) isn't automatically as good as building invariance into the network.
- The authors bound generalization error via covering numbers and show canonization sits between fully invariant models (best) and non-invariant baselines (worst).
- Whether canonization reaches the good end of that spectrum depends on the regularity (smoothness) of the canonization map.
- For point clouds, lexicographic sorting has exponential covering-number growth, but Hilbert curve canonization has polynomial growth — the first rigorous theory for why Hilbert serialization helps.
- Experiments on point-cloud tasks confirm the theoretical hierarchy.
Relevance
Not directly related to my persona/midtraining work — this is about symmetry in geometric data (point clouds), not language-model behavior or training interventions.
Abstract
arXiv:2605.11008v1 Announce Type: new Abstract: While invariant architectures are standard for processing symmetric data, there is growing interest in achieving invariance by applying group averaging or canonization to non-invariant backbones. However, the theoretical generalization properties of these alternative strategies remain poorly understood. We introduce a theoretical framework to analyze the generalization error of these methods by bounding their covering numbers. We establish a rigorous generalization hierarchy: the error bounds of canonized models are at best equal to the error bounds of structurally invariant and group-averaged models, and at worst equal to the bounds of non-invariant baselines. Furthermore, we show that there exist optimal canonizations which attain the optimal error bounds, and poor canonizations which attain the non-invariant error bounds, and that this depends on the regularity of the canonization. Finally, applying this framework to permutation groups in point cloud processing, we rigorously prove that the covering number of lexicographical sorting grows exponentially with point cloud dimension, whereas Hilbert curve canonization guarantees polynomial growth. This provides the first formal theoretical justification for the empirical success of Hilbert curve serialization in state-of-the-art point cloud architectures. We conclude with experiments that support our theoretical claims. Code is available at https://github.com/yonatansverdlov/Canonization