Extending Kernel Trick to Influence Functions
Authors: Zhenhuan Sun, Shahrokh Valaee
Summary
The authors present a "dual representation" of influence functions where computational complexity scales with dataset size rather than model size, making it efficient when models are large relative to datasets. This alternative can estimate how removing a data point changes parameters, outputs, or loss, but is limited to "linearizable models" (models whose behavior can be approximated by their linearization throughout training) and requires materializing a matrix whose size grows with output dimension times dataset size.
Main takeaways:
- Standard influence functions scale with model size; the dual representation scales with dataset size instead.
- Efficient alternative when model size is large relative to dataset size.
- Can estimate changes in parameters, outputs, and loss due to data point removal.
- Only works for linearizable models (models approximable by their linearization during training).
- Requires materializing a matrix of size (output dimension × dataset size), which can be a memory bottleneck.
Relevance
Potentially relevant to my work on finding prompts corresponding to narrow finetuning — influence functions can help understand what training data induced a behavior change. The dual representation could be useful if I'm working with large models and small behavioral-installation datasets, though the linearizability requirement might be limiting for LLMs.
Abstract
arXiv:2605.11239v1 Announce Type: cross Abstract: In this paper, we present a dual representation of the influence functions, whose computational complexity scales with dataset size rather than model size. Both analytically and experimentally, we show that this representation can be an efficient alternative to the original influence functions for estimating changes in parameters, model outputs and loss due to data point removal, when model size is large relative to dataset size, or when evaluating the original influence functions in parameter space is infeasible. The dual representation, however, is limited to linearizable models, which are models whose behavior can be approximated by their linearizations throughout training, and requires materializing a matrix, whose size grows with the product of model output dimension and dataset size.