SURGE: Surrogate Gradient Adaptation in Binary Neural Networks
Authors: Haoyu Huang, Boyu Liu, Linlin Yang et al.
Summary
SURGE tackles the gradient-mismatch problem in training binary neural networks (BNNs), where weights and activations are constrained to ±1. The standard Straight-Through Estimator (STE) approximates the gradient of the non-differentiable sign function with a surrogate, but this causes gradient mismatch and information loss. SURGE introduces a learnable gradient-compensation framework: for each binarized layer, it adds a parallel full-precision auxiliary branch and decomposes the output so gradients flow through both paths. The full-precision branch estimates higher-order gradient terms beyond STE's first-order approximation, and an adaptive scaler balances the two gradient streams. Experiments on image classification, object detection, and language tasks show SURGE outperforms state-of-the-art BNN training methods.
Main takeaways:
- BNN training relies on gradient approximations because the sign function is non-differentiable; STE and its variants suffer from gradient mismatch and fixed-range clipping.
- SURGE adds a parallel full-precision branch for each binarized layer, allowing gradients to flow through both binarized and full-precision paths.
- The Dual-Path Gradient Compensator (DPGC) decomposes outputs to decouple gradient flow; the full-precision branch captures components STE's first-order approximation misses.
- An Adaptive Gradient Scaler (AGS) dynamically balances gradient contributions from the two branches via norm-based scaling.
- SURGE achieves state-of-the-art results on image classification, object detection, and language-understanding benchmarks.
Relevance
Not directly related to my persona/midtraining work — SURGE is about gradient estimation in binary neural networks (extreme quantization), not about behavioral installation, fine-tuning pathways, or persona geometry in language models.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses bias.
Abstract
arXiv:2605.10989v1 Announce Type: new Abstract: The training of Binary Neural Networks (BNNs) is fundamentally based on gradient approximation for non-differentiable binarization operations (e.g., sign function). However, prevailing methods including the Straight-Through Estimator (STE) and its improved variants, rely on hand-crafted designs that suffer from gradient mismatch problem and information loss induced by fixed-range gradient clipping. To address this, we propose SURrogate GradiEnt Adaptation (SURGE), a novel learnable gradient compensation framework with theoretical grounding. SURGE mitigates gradient mismatch through auxiliary backpropagation. Specifically, we design a Dual-Path Gradient Compensator (DPGC) that constructs a parallel full-precision auxiliary branch for each binarized layer, decoupling gradient flow via output decomposition during backpropagation. DPGC enables bias-reduced gradient estimation by leveraging the full-precision branch to estimate components beyond STE's first-order approximation. To further enhance training stability, we introduce an Adaptive Gradient Scaler (AGS) based on an optimal scale factor to dynamically balance inter-branch gradient contributions via norm-based scaling. Experiments on image classification, object detection, and language understanding tasks demonstrate that SURGE performs best over state-of-the-art methods.