Distributional Reinforcement Learning via the Cramér Distance

Authors: Vanya Aziz, Ivo Nowak, E. M. T Hendrix

arXiv · PDF

Summary

The authors introduce C-DSAC, a reinforcement learning algorithm that combines the Soft Actor-Critic approach with distributional RL (learning a full distribution over future rewards instead of just an average). They use the squared Cramér distance to train the model and show it beats standard SAC and other distributional methods on robotics benchmarks, especially in complex environments. The key insight is that C-DSAC does "confidence-driven" updates: when the target distribution has high variance (low confidence), the model makes more conservative updates, reducing the impact of overestimated values.

Main takeaways:

• Represents state-action values as distributions rather than single numbers, capturing uncertainty about future rewards
• Uses the Cramér distance (a way to measure how different two probability distributions are) for learning
• High-variance (uncertain) target distributions automatically lead to smaller, more conservative model updates
• Outperforms baseline SAC and other distributional methods, with the gap widening in high-complexity robotics tasks
• The confidence-driven update mechanism helps prevent overly optimistic value estimates from corrupting learning

Relevance

Tangential — only relevant if I explore RL-based approaches to behavioral installation or persona shaping. The confidence-driven update mechanism could inspire thinking about uncertainty-aware fine-tuning, but this is primarily about value-based RL in continuous control, not language model behavior.

Threat model

Potential threat/caveat for clean result "Training a [ZLT] persona-marker into Qwen-2.5-7B doesn't increase system-prompt attention at the marker timestep — base Qwen on identical tokens attends the same way (LOW confidence)": this item discusses benchmark.

Abstract

arXiv:2605.08104v1 Announce Type: new Abstract: This paper explores the application of the Soft Actor-Critic (SAC) algorithm within a Distributional Reinforcement Learning setting and introduces an implementation of such algorithm named Cram'er-based Distributional Soft Actor-Critic (C-DSAC). The novel approach employs distributional reinforcement learning to represent state-action values, and minimizes the squared Cram'er distance for learning the distribution. Empirical results across various robotic benchmarks indicate that our algorithm surpasses the performance of baseline SAC and contemporary distributional methods, with the performance advantage becoming increasingly pronounced in high-complexity environments. To explain the efficiency of the new approach, we conduct an analysis showing that its superior performance is partly due to \textit{confidence-driven} Q-value updates: High-variance target distributions (low confidence in target) lead to more conservative model updates, thereby attenuating the impact of overestimated values. This work deepens the understanding of distributional reinforcement learning, offering insights into the algorithmic mechanisms governing convergence and value estimation.