Multi-Objective Constraint Inference using Inverse reinforcement learning
Authors: Syed Ihtesham Hussain Shah, Floris den Hengst, Aneta Lisowska et al.
Summary
The authors tackle constraint inference from demonstrations when different experts have different objectives—existing methods assume all demonstrations come from experts with identical goals, which is unrealistic. Their MOCI framework jointly learns shared safety constraints (rules everyone follows) and individual preferences (what makes each expert different) from heterogeneous expert trajectories using inverse reinforcement learning. On a grid-world benchmark, MOCI significantly outperforms baselines at predicting behavior while maintaining competitive computational efficiency.
Main takeaways:
- Existing constraint-inference methods assume homogeneous demonstrations (all experts share identical objectives), which limits their practical applicability
- MOCI extracts both shared constraints (safety boundaries everyone respects) and individual preferences (what makes experts different) from the same set of demonstrations
- The framework can learn from diverse and potentially conflicting behaviors exhibited by different experts
- Empirically outperforms existing baselines in predictive accuracy on a grid-world benchmark
- Maintains competitive computational efficiency despite handling the more complex heterogeneous setting
Relevance
Not directly related to my persona work, though there's a conceptual parallel: I'm studying how different personas (behavioral modes) are installed and conditioned, while this is about inferring what constraints and preferences underlie different expert behaviors. The notion of separating shared vs. individual components is loosely related to understanding what's shared across personas versus what's persona-specific.
Abstract
arXiv:2605.06951v1 Announce Type: new Abstract: Constraint inference is widely considered essential to align reinforcement learning agents with safety boundaries and operational guidelines by observing expert demonstrations. However, existing approaches typically assume homogeneous demonstrations (i.e., generated by a single expert or multiple experts with identical objectives). They also have limited ability to capture individual preferences and often suffer from computational inefficiencies. In this paper, we introduce Multi-Objective Constraint Inference (MOCI), a novel framework designed to jointly extract shared constraints and individual preferences from heterogeneous expert trajectories, where multiple experts pursue different objectives. MOCI effectively models and learns from diverse, and potentially conflicting, behaviors. Empirical evaluations demonstrate that MOCI significantly outperforms existing baselines, achieving improved predictive performance, and maintaining competitive computational efficiency on a standard grid-world benchmark. These results establish MOCI as an accurate, flexible, and computationally practical approach for real-world constraint inference and preference learning tasks.