HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model

Authors: Noam Kayzer, Dan Revital, Ori Bar Joseph et al.

arXiv · PDF

Summary

Hebatron is a Hebrew-specialized language model built on NVIDIA's sparse Mixture-of-Experts architecture (Nemotron-3). The authors use a three-phase "easy-to-hard" curriculum during training with continuous anti-forgetting anchoring (preventing the model from losing earlier knowledge), then supervised fine-tuning on 2 million bilingual Hebrew-English samples. The curriculum ordering alone yields a 3-point benchmark improvement over training in reverse order. Despite activating only 3B parameters per forward pass (in a 30B total parameter model), Hebatron achieves competitive Hebrew reasoning performance with ~9x higher inference throughput.

Main takeaways:

• First open-weight Hebrew-specialized Mixture-of-Experts model with native long-context support (up to 65,536 tokens).
• Three-phase easy-to-hard curriculum with anti-forgetting anchoring improves Hebrew reasoning by 3 points over reversed training order.
• Achieves 73.8% on Hebrew reasoning benchmarks, outperforming DictaLM-3.0-24B-Thinking (68.9%).
• Only 3B active parameters per forward pass across 30B total, giving ~9x higher throughput than dense models.
• Model weights are released openly for Hebrew and Semitic NLP research.

Relevance

Not directly related to my persona/conditional behavior work. The curriculum design and anti-forgetting anchoring are tangentially interesting for midtraining—they're trying to install Hebrew capability without destroying English, similar to how I'm trying to install markers/personas without collapsing geometry or causing leakage.

Threat model

Potential threat/caveat for clean result "Stretching turn count, completion length, or system-prompt length at train time fails to amplify marker uptake; the longest system prompt instead leaks across bystander personas (LOW confidence)": this item discusses benchmark.

Abstract

arXiv:2605.11255v1 Announce Type: new Abstract: We present Hebatron, a Hebrew-specialized open-weight large language model built on the NVIDIA Nemotron-3 sparse Mixture-of-Experts architecture. Training employs a three-phase easy-to-hard curriculum with continuous anti-forgetting anchoring, followed by supervised fine-tuning on 2 million bilingual Hebrew--English samples. The curriculum ordering alone yields a 3-point aggregate benchmark gain over the reversed configuration. Hebatron achieves a Hebrew reasoning average of 73.8%, outperforming DictaLM-3.0-24B-Thinking (68.9%) and remaining competitive with Gemma-3-27B-IT on GSM8K-HE and Israeli Trivia, while activating only 3B parameters per forward pass across a 30B-parameter model, delivering approximately 9 times higher inference throughput at native context lengths up to 65,536 tokens. To our knowledge, this is the first language-specific adaptation of the Nemotron-3 architecture for any target language, and the first open-weight Hebrew-specialized MoE model with native long-context support. Model weights are released openly to support further research in Hebrew and Semitic-language NLP.