EPS Dashboard

Authors: M7 QA

Summary

This internal QA document catalogs failure modes that arise when Sagan (the research system) generates a clean result only after an artifact row has been verified. The authors propose a set of benchmark checks for the artifact-verification step, review comment templates for clean results, and negative-control patterns to catch spurious conclusions. The motivation is to reduce the risk that a passing artifact verification masks hidden problems in how findings are recorded and summarized.

Main takeaways:

Identifies specific failure modes in the artifact-verification → clean-result pipeline that can silently corrupt reported findings.
Proposes concrete benchmark checks to stress-test the verification step before a clean result is accepted.
Recommends standardized review-comment templates to make caveat recording more consistent across clean results.
Argues that negative controls should be built into the clean-result review process, not added post-hoc.
Frames artifact verification as a distinct quality-assurance layer from statistical significance checking.

Relevance

My work produces clean results (e.g., the LoRA persona-leakage results, backdoor-trigger findings, and bystander-spill experiments) that all pass through the artifact-verification step this document critiques — the failure modes cataloged here could directly affect how confidently I should trust my own MODERATE/LOW confidence labels.

Threat model

If the artifact-verification step has the failure modes described here, some of my MODERATE-confidence clean results (e.g., the 23.5% paired-marker leakage or the bystander-spill rates) may have passed verification for procedural reasons rather than genuine replication, warranting a re-check against the proposed benchmark suite.

Abstract

This paper studies failure modes and caveats when Sagan creates a clean result only after verifying an artifact row. It proposes benchmark checks for artifact verification, clean-result review comments, and negative controls.