Couple to Control: Joint Initial Noise Design in Diffusion Models

Authors: Jing Jia, Liyue Shen, Guanyang Wang

arXiv · PDF

Summary

The authors argue that diffusion models don't need to start from independent Gaussian noise for each image in a batch — instead, you can "couple" the initial noises (each remains marginally Gaussian, so the model sees the same distribution per sample, but samples are no longer independent). This reframes noise control from picking individual seeds to designing dependence structure across a gallery. Repulsive Gaussian coupling improves gallery diversity on SD1.5, SDXL, and SD3 without adding sampling cost, matching or outperforming recent noise-optimization baselines.

Main takeaways:

• Standard diffusion generation uses independent Gaussian noise per sample, but this is just one choice — you can design coupled noise with chosen dependence structure.
• Each noise remains marginally standard Gaussian, so pretrained models see the same single-sample input distribution.
• Repulsive coupling (samples push away from each other) improves gallery diversity while preserving prompt alignment and image quality.
• Matches or beats recent noise-optimization baselines on diversity metrics at the same sampling cost as independent generation.
• Subspace couplings enable fixed-object background generation with diverse, natural backgrounds.

Relevance

Not directly related to my persona/midtraining work — this is about diffusion model noise initialization for image generation, not about language model behavioral installation or persona geometry.

Abstract

arXiv:2605.11311v1 Announce Type: cross Abstract: Diffusion models typically generate image batches from independent Gaussian initial noises. We argue that this independence assumption is only one choice within a broader class of valid joint noise designs. Instead, one can specify a coupling of the initial noises: each noise remains marginally standard Gaussian, so the pretrained diffusion model receives the same single-sample input distribution, while the dependence across samples is chosen by design. This reframes initial-noise control from selecting or optimizing individual seeds to designing the dependence structure of a multi-sample gallery. This view gives a general framework for initial-noise design, covering several existing methods as special cases and leading naturally to new coupled-noise constructions. Coupled noise can improve generation on its own without adding sampling cost, and it is flexible enough to serve as a structured initialization for optimization-based pipelines when additional computation is available. Empirically, repulsive Gaussian coupling improves gallery diversity on SD1.5, SDXL, and SD3 while largely preserving prompt alignment and image quality. It matches or outperforms recent test-time noise-optimization baselines on several diversity metrics at the same sampling cost as independent generation. Subspace couplings also support fixed-object background generation, producing diverse, natural backgrounds compared with specialized inpainting baselines, with a tunable trade-off in foreground fidelity.