Constraint-Data-Value-Maximization: Utilizing Data Attribution for Effective Data Pruning in Low-Data Environments
Authors: Danilo Brajovic, David A. Kreplin, Marco F. Huber
Summary
The authors tackle data pruning in low-data environments where you want to keep only the most valuable training examples. Existing Shapley-based data valuation methods don't work well when you prune aggressively—they optimize for total influence but can leave you with a dataset dominated by a few high-leverage outliers. They propose CDVM (Constraint-Data-Value-Maximization), which adds a penalty to prevent any single test case from being over-represented, yielding more robust performance when only a small fraction of data remains.
Main takeaways:
- Shapley-based data values underperform when pruning to very small dataset sizes because they can over-index on high-leverage outliers.
- CDVM frames pruning as constrained optimization: maximize total data influence while penalizing excessive per-test contributions.
- On the OpenDataVal benchmark, CDVM delivers strong performance and competitive runtime, especially in low-data regimes.
- The approach is about balancing dataset diversity and representativeness, not just total influence.
- Useful for scenarios where labeled data is scarce and you need to carefully select which examples to keep for training.
Relevance
Tangential—only relevant if I want to prune training data for persona or midtraining experiments. Could inform sample selection if I'm trying to isolate which training examples are responsible for specific persona behaviors.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses limitation, benchmark.
Abstract
arXiv:2605.11312v1 Announce Type: new Abstract: Attributing model behavior to training data is an evolving research field. A common benchmark is data removal, which involves eliminating data instances with either low or high values, then assessing a model's performance trained on the modified dataset. Many existing studies leverage Shapley-based data values for this task. In this paper, we demonstrate that these data values are not optimally suited for pruning low-value data when only a limited amount of data remains. To address this limitation, we introduce the Constraint-Data-Value-Maximization (CDVM) approach, which effectively utilizes data attributions for pruning in low-data scenarios. By casting pruning as a constrained optimization that both maximizes total influence and penalizes excessive per-test contributions, CDVM delivers robust performance when only a small fraction of the data is retained. On the OpenDataVal benchmark, CDVM shows strong performance and competitive runtime.