EPS Dashboard

Authors: Georgios Batzolis, Mark Girolami, Luca Ambrogioni

Summary

The authors propose a diffusion language model that represents text as a continuous diffusion process over fixed-width binary bitstreams, aiming to close the gap with autoregressive models. Each semantic token is represented as an analog bit sequence, and the model uses a matched-filter residual parameterization to separate contextual learning from the independent-bit posteriors. Crucially, they introduce a stochastic sampler that applies Langevin-type corrections gated by the entropy-rate profile—concentrating randomness in high-information regions while remaining nearly deterministic elsewhere. On LM1B, their 130M-parameter model reaches a generative perplexity of 59.76 (matching the autoregressive reference) using 256 function evaluations; on OpenWebText, it achieves 27.06 perplexity with 4× fewer steps than prior 1024-step baselines. Bitstream diffusion also removes the vocabulary-size scaling bottleneck, predicting O(log V) bitwise logits instead of O(V) token logits.

Main takeaways:

Represents text as a diffusion process over continuous binary bitstreams (fixed-width analog bits) rather than discrete tokens.
Stochastic sampler gates Langevin corrections by entropy rate, concentrating randomness where information is high and being deterministic elsewhere.
On LM1B: 130M-param model hits generative perplexity 59.76 (matching autoregressive baseline) in 256 neural function evaluations.
On OpenWebText: achieves perplexity 27.06 with 4× fewer steps than prior 1024-step diffusion baselines.
Removes O(V) vocabulary scaling bottleneck by predicting O(log V) bitwise logits, reducing memory and increasing throughput.

Relevance

Not directly related to my persona or midtraining work—this is about diffusion-based language modeling architecture. Could be tangentially relevant if I ever explored whether continuous-representation models (like bitstream diffusion) respond differently to persona installation than discrete autoregressive models.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation, benchmark.

Abstract

arXiv:2605.07013v1 Announce Type: new Abstract: Diffusion language models (DLMs) promise parallel, order-agnostic generation, but on standard benchmarks they have historically lagged behind autoregressive models in sample quality and diversity. Recent continuous flow and diffusion approaches over token embeddings have narrowed this gap, suggesting continuous state spaces are highly effective for language. In this work, we further close the autoregressive gap by modeling text as a continuous diffusion process over fixed-width binary bitstreams. Our approach represents semantic tokens as analog bit sequences and utilizes a matched-filter residual parameterization to isolate contextual learning from analytic independent-bit posteriors. Crucially, we adopt a stochastic sampler that applies Langevin-type corrections gated by the entropy-rate profile, automatically concentrating stochasticity in high-information regions while remaining nearly deterministic elsewhere. On the One Billion Word Benchmark (LM1B), our 130M-parameter bitstream model reaches a generative perplexity ($\GenPPL$) of $59.76$ at matched real-data entropy ($4.31$) using 256 neural function evaluations (NFEs), decisively outperforming prior DLM baselines and reaching the autoregressive reference. On OpenWebText (OWT), our stochastic sampler establishes a new continuous-DLM Pareto frontier, achieving $\GenPPL=27.06$ at an entropy of $5.26$ using $4\times$ fewer steps than previous 1024-NFE baselines. As an additional architectural benefit, bitstream diffusion removes the $\mathcal{O}(V)$ vocabulary scaling bottleneck shared by standard DLMs. By predicting $\mathcal{O}(\log V)$ bitwise logits via semantic bit-patching, our model yields a reduced memory footprint and higher throughput, demonstrating a scalable paradigm for language generation as vocabulary sizes grow.