Metropolis-Adjusted Diffusion Models
Authors: Kevin H. Lam, Tyler Farghly, Christopher Williams et al.
Summary
Diffusion models for image generation are biased because of discretization and imperfect score function estimates. Existing corrector steps (like unadjusted Langevin) don't fully fix this. The authors propose using Metropolis-Hastings or Barker accept-reject steps to eliminate the bias from discretization. Since the usual target density ratio isn't available, they show how to compute correct acceptance probabilities using the score function instead. They introduce the first exact correction via a two-coin Bernoulli factory and a practical approximation using Simpson's rule that's very accurate and nearly free computationally. Experiments show improved sample quality (better FID scores) on image datasets.
Main takeaways:
- Standard corrector steps in diffusion models (like unadjusted Langevin) are themselves biased due to discretization
- Metropolis-Hastings or Barker corrections can remove this bias, but require a target density ratio that's unavailable
- New methods compute acceptance probabilities using only the score function
- An exact correction uses a Bernoulli factory; a practical approximation uses Simpson's rule with order 5/2 accuracy
- Empirical results show consistent improvements in image quality (FID) on synthetic and real datasets
Relevance
Not related to my language model behavioral work—this is about correcting bias in image-generation diffusion models, not about personas, prompts, or steering vectors.
Abstract
arXiv:2605.09654v1 Announce Type: new Abstract: Sampling from score-based diffusion models incurs bias due to both time discretisation and the approximation of the score function. A common strategy for reducing this bias is to apply corrector steps based on the unadjusted Langevin algorithm (ULA) at each noise level within a predictor-corrector framework. However, ULA is itself a biased sampler, as it discretises a continuous diffusion process. In this work, we consider adjusted Langevin correctors that employ Metropolis--Hastings (MH) or Barker's accept-reject steps to correct for this bias. Since the target density ratio typically required by MH-based algorithms is unavailable, we propose methods that instead utilise the score function to compute the correct acceptance probability. We introduce the first exact method for adjusting Langevin corrections in diffusion models, based on a two-coin Bernoulli factory algorithm. We also propose an efficient approximation based on Simpson's rule that achieves accuracy of order $5/2$ in the step size at near-zero marginal cost. We demonstrate that these procedures improve sample quality on both synthetic and image datasets, yielding consistent gains in Fr'echet Inception Distance (FID) on the latter.