Quantifiable Uncertainty: A Stochastic Consensus Multi-Agent RAG Framework for Robust Malware Detection
Authors: ElMouatez Billah Karbab
Summary
MAGMA is a Retrieval-Augmented Generation framework for malware detection that decouples analysis into semantic code retrieval and probabilistic verification. The system uses dual-stream embeddings over assembly and pseudo-code to isolate critical functions, then employs multiple reasoning agents with non-deterministic sampling to produce two metrics: Function Evidence Strength and Evidence Conflict Score (Shannon entropy of predictions). High entropy serves as a signal for structural ambiguity, enabling a reject-option policy that achieves 98.4% detection rate.
Main takeaways:
- Using ensemble entropy as a proxy for epistemic uncertainty enables the system to flag ambiguous cases rather than making unreliable predictions
- The approach addresses evasion attacks by expressing uncertainty, unlike monolithic classifiers
- Dual-stream embedding helps separate decision-critical functions from dead code noise
Relevance
This paper demonstrates using ensemble disagreement (entropy) to quantify model uncertainty and implement reject options for security-critical decisions.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses limitation, evaluation.
Abstract
arXiv:2605.08385v1 Announce Type: new Abstract: While contemporary deep learning malware detectors define a dominant defense paradigm, their sophistication also exposes them to novel structural evasion attacks, a limitation we attribute to their inherent inability to express epistemic uncertainty. To address this challenge, we present MAGMA, a Retrieval-Augmented Generation (RAG) framework that decouples malware analysis into semantic code retrieval and probabilistic verification. In contrast to monolithic classifiers, MAGMA employs a dual-stream embedding scheme over assembly and pseudo-code representations to isolate Decision-Critical Functions (DCFs) from the noise of dead code. We further introduce a Stochastic Consistency Ensemble, in which multiple instances of the same reasoning agent independently evaluate the retrieval set under non-deterministic sampling. From this ensemble, we derive two complementary metrics: Function Evidence Strength (FES), a weighted aggregation of retrieval confidence, and the Evidence Conflict Score (ECS), defined as the Shannon entropy of the ensemble's predictive distribution. We show that elevated ECS values serve as an effective proxy for structural ambiguity, enabling the system to implement a principled ``reject-option'' policy. Extensive evaluation demonstrates that MAGMA achieves a 98.4% detection rate, substantially exceeding existing solutions.