Fast Training of Mixture-of-Experts for Time Series Forecasting via Expert Loss Integration
Authors: Btissame El Mahtout, Florian Ziel
Summary
The authors build a mixture-of-experts (MoE) model for time series forecasting that trains experts to specialize by giving each expert its own loss function—not just a global prediction loss, but also expert-specific errors that guide which expert learns what patterns. They combine this with incremental online updates so you can adapt the model without retraining from scratch every time, cutting computational cost. Tests on economic, tourism, and energy datasets show the approach beats standard statistical methods and modern neural architectures (Transformers, WaveNet) in both accuracy and efficiency.
Main takeaways:
- Each expert gets its own loss signal during training, encouraging specialization beyond what a shared global loss would provide
- Online learning lets you update the gating network and expert weights incrementally, avoiding full retraining
- Outperforms statistical baselines and Transformer/WaveNet models on multiple real-world forecasting tasks
- Ablation studies confirm that the expert-specific loss design is doing real work—not just a hyperparameter tweak
Relevance
Not directly related to my persona/midtraining work—this is about time-series forecasting with MoE architectures. Included because the idea of expert-specific loss signals vaguely parallels conditioning behavior on different contexts, but the domain and methods are far from language-model persona installation.
Abstract
arXiv:2605.10330v1 Announce Type: new Abstract: We propose a novel adaptive Mixture-of-Experts (MoE) framework for time series forecasting that enhances expert specialization by incorporating expert-specific loss information directly into the training process. Notably, the overall objective comprises the base forecasting loss and expert-specific losses, allowing expert-level prediction errors to jointly shape training alongside the global forecasting loss. This framework is further combined with a partial online learning strategy, enabling incremental updates of both the gating mechanism and expert parameters. This approach significantly reduces computational cost by eliminating the need for repeated full model retraining. By integrating expert-level loss awareness with efficient online optimization, the proposed method achieves improved learning efficiency while maintaining strong predictive performance. Empirical results across economic, tourism, and energy datasets with varying frequencies demonstrate that the proposed approach generally outperforms both statistical methods and state-of-the-art neural network models, such as Transformers and WaveNet, in forecasting accuracy and computational efficiency. Furthermore, ablation studies confirm the effectiveness of the expert-specific loss integration strategy, highlighting its contribution to enhancing predictive performance.