OracleTSC: Oracle-Informed Reward Hurdle and Uncertainty Regularization for Traffic Signal Control
Authors: Darryl Jacob, Xinyu Liu, Muchao Ye et al.
Summary
The authors built a traffic-light control system using a language model that gets reinforcement-learning feedback from traffic simulators. Standard RL struggled because most actions barely change congestion (sparse, delayed rewards), so they added two tricks: a "reward hurdle" that filters out weak feedback signals by subtracting a threshold, and "uncertainty regularization" that pushes the model to be consistent across multiple attempts at the same decision. An 8-billion-parameter LLaMA model trained this way cut travel time by 75% and transferred to a different intersection layout without retraining.
Main takeaways:
- Language models doing RL on traffic control are unstable because most green-light timing changes produce tiny, delayed improvements in congestion
- Subtracting a calibrated threshold from rewards (the "hurdle") filters out noise and focuses learning on actions that matter
- Forcing the model to give consistent answers across sampled outputs (uncertainty regularization) stabilizes training
- A policy learned on one intersection generalized to a structurally different intersection (17% lower travel time, 39% shorter queues) with no fine-tuning
- The system keeps interpretability because it outputs natural-language explanations for its decisions
Relevance
Tangential—only loosely relevant in that it's another case of behavioral fine-tuning (RL here vs. SFT in my work) where reward signal strength and consistency matter, but traffic control has nothing to do with personas, markers, or conditional behavior in chat models.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.
Abstract
arXiv:2605.08516v1 Announce Type: new Abstract: Transparent decision-making is essential for traffic signal control (TSC) systems to earn public trust. However, traditional reinforcement learning-based TSC methods function as black boxes with limited interpretability. Although large language models (LLMs) can provide natural language reasoning, reinforcement finetuning for TSC remains unstable because feedback is sparse and delayed, while most actions produce only marginal changes in congestion metrics. We introduce OracleTSC, which stabilizes LLM-based TSC through two mechanisms: (1) a reward hurdle mechanism that filters weak learning signals by subtracting a calibrated threshold from environmental rewards, and (2) uncertainty regularization that maximizes the probability of the selected response to encourage consistent decisions across sampled outputs. Experiments on the LibSignal benchmark show that OracleTSC enables a compact LLaMA3-8B model to substantially improve traffic efficiency, achieving a 75% reduction in travel time and a 67% decrease in queue length compared with the pretrained baseline while preserving interpretability through natural language explanations. OracleTSC also demonstrates strong cross-intersection generalization: a policy trained on one intersection transfers to a structurally different intersection with 17% lower travel time and 39% lower queue length without additional finetuning. These results suggest that uncertainty-aware reward shaping can improve the stability and effectiveness of reinforcement fine-tuning for TSC.