Seeing the Needle in the Haystack: Towards Weakly-Supervised Log Instance Anomaly Localization via Counterfactual Perturbation
Authors: Yutszyuk Wong, Wentai Wu, Yuen-Ying Yeung et al.
Summary
The authors tackle anomaly detection in system logs when you only have coarse labels (e.g., "this batch of logs contains an anomaly somewhere") rather than labels on individual log entries. Their LogMILP framework uses multi-instance learning — treating a bag of logs as the unit of supervision — combined with prototype-guided modeling and counterfactual perturbations to identify which specific log entries are anomalous. This lets operators pinpoint the exact problematic events without expensive instance-level annotations.
Main takeaways:
- Trains on bag-level labels ("anomaly present in this collection") but produces instance-level localization ("line 47 is the culprit").
- Uses prototypes (representative examples) and counterfactual perturbations ("if I remove this entry, does the bag stop looking anomalous?") to guide the model to the critical log entries.
- Achieves competitive detection performance while improving localization reliability on three public log datasets.
- Code is open-sourced for practitioners working with large-scale networked systems.
Relevance
Not directly related to my persona/midtraining work — this is a log-anomaly detection paper for system operations, with no obvious connection to language model behavior or prompt-based persona installation.
Abstract
arXiv:2605.10988v1 Announce Type: new Abstract: Log anomaly detection is a critical task for system operations and security assurance. However, in networked systems at scale, log data are generated at massive scale while instance-level annotations are prohibitively expensive, posing great difficulties to fine-grained anomaly localization. To address this challenge, we propose LogMILP (Log anomaly localization based on Multi-Instance Learning enhanced by prototypes and Perturbation), a weakly supervised framework that enables both bag-level anomaly detection and instance-level anomaly localization using only bag-level labels. Our method guides the model to pinpoint the critical log entries using prototype-guided structural modeling with counterfactual perturbation consistency regularization, thereby improving localization reliability and interpretability under coarse-grained supervision. Experimental results on three public datasets demonstrate that LogMILP achieves competitive detection performance while yielding significantly more reliable instance-level localization. Our code is open-sourced at https://github.com/YUK1207/LogMILP.