Rethinking LLMOps for Fraud and AML: Building a Compliance-Grade LLM Serving Stack
Authors: Prathamesh Vasudeo Naik, Naresh Dintakurthi, Yue Wang
Summary
The authors build a compliance-grade LLM serving stack for fraud and AML workloads, which have prefix-heavy prompts (reusable policy text, risk taxonomies) and schema-constrained outputs (JSON labels). They combine vLLM runtime tuning, PagedAttention, automatic prefix caching, multi-adapter serving, adapter-aware batching, and speculative decoding. On public-synthetic AML datasets, workload-aware tuning improved throughput from 612–650 to 3,600 requests/hour, reduced P99 latency from 31–38 seconds to 6.4–8.7 seconds, and increased GPU utilization from 12% to 78%.
Main takeaways:
- Fraud/AML compliance prompts are prefix-heavy with reusable policy instructions and schema-constrained outputs, making KV-cache efficiency and prefix reuse critical.
- Stack combines vLLM-style runtime tuning, PagedAttention, automatic prefix caching, multi-adapter serving, and speculative decoding.
- Workload-aware tuning improved throughput 5–6×, reduced P99 latency 4–5×, and raised GPU utilization from 12% to 78%.
- Includes LLM-as-judge quality gate with deterministic compliance checks and multi-judge rubric scoring.
- Demonstrates that regulated LLM workloads require specialized serving infrastructure tuned to their prompt structure and output constraints.
Relevance
Not directly related to my persona/midtraining work—included because prefix caching and adapter-aware batching could be useful if I'm running many persona-finetuned adapters or testing marker-implantation across many system prompts at scale.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.
Abstract
arXiv:2605.11232v1 Announce Type: new Abstract: Fraud detection and anti-money-laundering (AML) compliance are high-value domains for large language models (LLMs), but their serving requirements differ sharply from generic chat workloads. Compliance prompts are often prefix-heavy, schema-constrained, and evidence-rich, combining reusable policy instructions, risk taxonomies, transaction or document context, and short structured outputs such as JSON labels or risk factors. These properties make prefix reuse, KV-cache efficiency, runtime tuning, model orchestration, and output validation first-order systems concerns. This paper introduces a workload-aware LLMOps stack for fraud and AML workloads using self-hosted open-weight models such as Meta Llama and Alibaba Qwen. The stack combines vLLM-style runtime tuning, PagedAttention, Automatic Prefix Caching, multi-adapter serving, adapter and prompt-length-aware batching, sleep/wake lifecycle management, speculative decoding, and optional prefill/decode disaggregation. To avoid exposing institution-specific data, the reproducibility track converts public synthetic AML datasets, including IBM AML and SAML-D, into prefix-heavy compliance prompts with reusable policy text, transaction evidence, typology definitions, and schema-constrained outputs. We also incorporate an LLM-as-judge quality gate using deterministic compliance checks, reference metrics, expert-adjudicated calibration data where available, and multi-judge rubric scoring. Across public-synthetic AML workloads and controlled serving benchmarks, workload-aware tuning improved throughput from 612-650 to 3,600 requests/hour, reduced P99 latency from 31-38 seconds to 6.4-8.7 seconds, and increased GPU utilization from 12% to 78%. These results show that regulated LLM performance is a workload-design, serving-optimization, and quality-gating problem, not only a model-selection problem.