EPS
← All batches·2605.11511

Post-ADC Inference: Valid Inference After Active Data Collection

topic: othertop score: 100released: 2026-05-13first surfaced: 2026-05-13arXivPDFthreats2026-05-13

Authors: Shuichi Nishino, Tomohiro Shiraishi, Teruyuki Katsuoka et al.

arXiv · PDF

Summary

The authors develop a framework for valid statistical inference when data are collected via active data collection (ADC)—e.g., Bayesian optimization or sequential model-based optimization—and then reused for a post-hoc inferential task (like testing whether a discovered setting is truly optimal). Standard inference fails because ADC preferentially samples regions the algorithm thinks are good, creating adaptive bias. The "post-ADC inference" framework corrects for both the bias from adaptive data collection and the bias from constructing the inferential target in a data-dependent way, providing valid p-values and confidence intervals.

Main takeaways:

  • Active data collection (e.g., Bayesian optimization) adaptively biases sampling toward promising regions, breaking standard statistical inference.
  • Standard p-values and confidence intervals are invalid when you reuse ADC data for post-hoc inference (e.g., "is this the true optimum?").
  • Post-ADC inference corrects for both the adaptive sampling bias and the data-dependent construction of the inferential target.
  • The method builds on selective inference and applies to a broad class of ADC processes (only assumes observation noise, not the black-box function or surrogate model).
  • Empirical results show valid inference for data collected by GP-UCB and tree-structured Parzen estimator (TPE) with correct coverage.

Relevance

Not directly related to my persona/midtraining work—this is about valid inference after adaptive data collection in optimization. Could be tangentially relevant if I'm thinking about adaptively collecting training data or hyperparameter tuning during fine-tuning, but the connection is weak.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses bias, evaluation.

Abstract

arXiv:2605.11511v1 Announce Type: new Abstract: The validity of statistical inference depends critically on how data are collected. When data gathered through active data collection (ADC) are reused for a post-hoc inferential task, conventional inference can fail because the sampling is adaptively biased toward regions favored by the collection strategy. This issue is especially pronounced in black-box optimization, where sequential model-based optimization (SMBO) methods such as the tree-structured Parzen estimator (TPE) and Gaussian process upper confidence bound (GP-UCB) preferentially concentrate evaluations in promising regions. We study statistical inference on actively collected data when the inferential target is constructed in a data-dependent manner after data collection. To enable valid inference in this setting, we propose post-ADC inference, a framework that accounts for the biases arising from both the active data collection process and the subsequent data-driven target construction. Our method builds on selective inference and provides valid $p$-values and confidence intervals that correct for both sources of bias. The framework applies to a broad class of ADC processes by imposing only assumptions on the observation noise, without requiring any assumptions on the underlying black-box function or the surrogate model used by the SMBO algorithm. Empirical results also show that post-ADC inference provides valid inference for data collected by GP-UCB and TPE.