HIDBench: Benchmarking Large Language Models for Host-Based Intrusion Detection
Authors: Danyu Sun, Jinghuai Zhang, Yuan Tian et al.
Summary
arXiv:2605. 21773v1 Announce Type: new Abstract: Recent benchmark efforts have advanced the evaluation of large language models (LLMs) in cybersecurity, including tasks such as penetration testing and vulnerability identification.
Relevance
Read next because HIDBench: Benchmarking Large Language Models for Host-Based Intrusion Detection overlaps with clean result "LoRA persona trained on alone emits at 23.5% when a co-trained partner learns ..., vs 0% control on Qwen2.5-7B-Instruct (MODERATE confidence)", clean result "Leakage rate is a usable signal for recovering trigger-shaped phrases on Gaperon-1125-1B without knowing the hidden trigger itself (MODERATE confidence)", clean result "Language-mismatch LoRA SFT on Qwen2.5-7B leaks the trained completion language into bystander directives the model was never trained on, absent under same-language SFT (LOW confidence)". Matching terms: strong, under, eval, line, rate, test, language, model. Source: arxiv cs.CR (Cryptography and Security).
Threat model
Potential threat/caveat for clean result "LoRA persona trained on alone emits at 23.5% when a co-trained partner learns ..., vs 0% control on Qwen2.5-7B-Instruct (MODERATE confidence)": this item discusses evaluation, benchmark.
Abstract
arXiv:2605.21773v1 Announce Type: new Abstract: Recent benchmark efforts have advanced the evaluation of large language models (LLMs) in cybersecurity, including tasks such as penetration testing and vulnerability identification. However, a critical cybersecurity task, namely intrusion detection from system logs, remains unexplored. In this work, we present a new benchmark to assess LLMs' capabilities in supporting host-based intrusion detection systems (HIDS). This task requires fine-grained reasoning over large-scale, noisy, and highly imbalanced system logs, where complex interactions between benign and malicious activities make reliable detection challenging. Our benchmark unifies three public system log datasets, DARPA-E3, DARPA-E5, and NodLink, and introduces a data construction pipeline that transforms raw host telemetry into LLM-compatible inputs, enabling systematic evaluation under realistic intrusion detection settings. Our evaluation of frontier LLMs reveals substantial performance gaps across datasets. While many models achieve high precision (often above 0.8) on simpler datasets, their performance degrades significantly as system logs become noisier and more complex, with MCC frequently dropping below 0.5 and false positive rates increasing sharply. We further analyze model behavior and identify distinct regimes, including conservative detectors with low false positive rates and over-sensitive models that generate excessive alerts. Overall, our results highlight that while LLMs show strong potential for HIDS, their effectiveness is highly sensitive to data complexity, and robust system design is essential for reliable deployment.