Minimax Rates and Spectral Distillation for Tree Ensembles

Authors: Binh Duc Vu, David S. Watson

arXiv · PDF

Summary

The authors study tree ensembles (random forests and gradient boosting) from a spectral perspective—looking at the eigenvalues and eigenvectors of the implicit kernel or smoother matrix these methods induce. They derive minimax-optimal convergence rates for random forest regression (showing the rate depends on how fast the kernel's eigenvalues decay) and use the spectral viewpoint to compress ensembles. For random forests, the leading eigenfunctions of the kernel capture the most important predictive directions; for GBMs, the leading singular vectors of the smoother matrix do the same. Training small nonlinear maps on these spectral features yields "distilled" models orders of magnitude smaller than the originals with competitive performance.

Main takeaways:

• Random forests and GBMs can be viewed through their induced kernels or smoother matrices; the eigenvalue decay of these operators governs statistical convergence rates.
• Under mild conditions, random forest regression achieves minimax-optimal rates.
• Leading eigenfunctions (for RFs) or singular vectors (for GBMs) capture the ensemble's dominant predictive structure.
• Training small models on these spectral representations compresses ensembles by orders of magnitude with minimal performance loss.
• The method outperforms existing pruning and rule-extraction techniques on real datasets.

Relevance

Not directly related to my persona/midtraining work—this is about compressing tree ensembles via spectral methods. Could be tangentially relevant if I'm thinking about how to compress or distill fine-tuned models, but the techniques are specific to tree-based methods, not transformers.

Abstract

arXiv:2605.11841v1 Announce Type: new Abstract: Tree ensembles such as random forests (RFs) and gradient boosting machines (GBMs) are among the most widely used supervised learners, yet their theoretical properties remain incompletely understood. We adopt a spectral perspective on these algorithms, with two main contributions. First, we derive minimax-optimal convergence for RF regression, showing that, under mild regularity conditions on tree growth, the eigenvalue decay of the induced kernel operator governs the statistical rate. Second, we exploit this spectral viewpoint to develop compression schemes for tree ensembles. For RFs, leading eigenfunctions of the kernel operator capture the dominant predictive directions; for GBMs, leading singular vectors of the smoother matrix play an analogous role. Learning nonlinear maps for these spectral representations yields distilled models that are orders of magnitude smaller than the originals while maintaining competitive predictive performance. Our methods compare favorably to state of the art algorithms for forest pruning and rule extraction, with applications to resource constrained computing.