Adaptive auditing of AI systems with anytime-valid guarantees

Authors: Siyu Zhou, Patrick Vossler, Venkatesh Sivaraman et al.

arXiv · PDF

Summary

The authors tackle a practical problem in AI auditing: when you're adaptively testing a model (deciding on-the-fly which cases to annotate based on what you've seen), classical statistics breaks down because you're peeking at the data and changing your plan mid-stream. They introduce a "testing by betting" framework using Safe Anytime-Valid Inference (SAVI) that lets auditors draw statistically rigorous conclusions from very small samples (sometimes just 20 observations) even when they're adaptively choosing what to test next. The framework sets up dueling hypotheses—the model claiming it has no failure modes below a threshold versus the auditor claiming they can find one—and proves these are asymptotically inverses, meaning passing a tough audit actually certifies robustness.

Main takeaways:

• Adaptive testing (choosing what to evaluate based on earlier results) is practical but violates classical statistics assumptions, making it hard to draw rigorous conclusions from small samples
• The "testing by betting" approach maintains statistical validity even when you change your sampling strategy mid-audit and work with only 10-50 test cases
• The framework formalizes two competing perspectives: the model's claim of no serious failures versus the auditor's claim they can find one
• If the auditor is sufficiently thorough, passing their audit provably certifies the model as globally robust (the two hypotheses become inverses)
• Empirically outperforms pre-specified (non-adaptive) testing and sometimes reaches valid conclusions with as few as 20 observations

Relevance

Not directly related to my persona/midtraining work—this is about auditing deployed systems for failure modes, not about understanding how behaviors are installed or conditioned during training. Might be useful if I ever need to rigorously test whether a persona or marker has leaked to unintended contexts.

Abstract

arXiv:2605.07002v1 Announce Type: new Abstract: A major bottleneck in characterizing the failure modes of generative AI systems is the cost and time of annotation and evaluation. Consequently, adaptive testing paradigms have gained popularity, where one opportunistically decides which cases and how many to annotate based on past results. While this framework is highly practical, its extreme flexibility makes it difficult to draw statistically rigorous conclusions, as it violates classical assumptions: the number of observations is typically limited (often 10 to 50 cases) and decisions regarding sampling and stopping are made in the midst of data collection rather than based a pre-specified rule. To characterize what statistical inferences can be drawn from highly adaptive audits, we introduce a hypothesis testing framework from two 'dueling' perspectives: (i) the model's null that asserts there is no failure mode with performance below a target threshold versus (ii) the auditor's null that asserts they have a sampling strategy that will uncover a failure mode. Leveraging Safe Anytime-Valid Inference (SAVI), we formalize the auditor as conducting 'testing by betting', which translates into simultaneous e-processes for testing the dueling null hypotheses. Furthermore, if the auditor is sufficiently powerful, we prove that these two hypotheses are asymptotically inverses of each other, in that passage of a stringent audit does in fact certify the AI system as being globally robust. Empirically, we demonstrate that our proposed testing procedures maintain anytime-valid type-I error control, outperform pre-specified testing methods, and can reach statistically rigorous conclusions sometimes with as few as 20 observations.