CPEMH: An Agentic Framework for Prompt-Driven Behavior Evaluation and Assurance in Foundation-Model Systems for Mental Health Screening

Authors: Giuliano Lorenzoni (University of Waterloo), Ivens Portugal (University of Waterloo), Paulo Alencar (University of Waterloo) et al.

arXiv · PDF

Summary

The authors built CPEMH, a system that uses multiple AI agents working together to automatically design, test, and choose prompts for mental-health screening tasks. Instead of manually tweaking prompts and hoping they work reliably, the framework orchestrates three specialized agents—one to coordinate, one to run inference, and one to evaluate—ensuring that prompt-driven behavior is stable and auditable. They tested it on depression screening from interview transcripts and found that modular orchestration helps keep foundation-model behavior predictable in sensitive clinical settings.

Main takeaways:

• Uses an "agentic" architecture where separate agents design prompts, run them, and measure their performance, all coordinated automatically.
• Focuses on behavioral assurance: making sure the model's responses are stable and traceable across different contexts, not just accurate once.
• Case study on depression screening shows the framework can stabilize model behavior in conversational clinical domains.
• Emphasizes simplicity over complexity—stability matters more than fancy architecture—and tracks F1, bias, and robustness as acceptance criteria.
• Provides an engineering methodology for controlling prompt-driven variability in foundation models applied to real-world sensitive tasks.

Relevance

Directly relevant to my work on finding prompts that correspond to specific behaviors (installation-path equivalence, prompt vs fine-tuning). Their framework for systematically searching and evaluating prompts that drive stable behaviors could inform methods to reverse-engineer prompts from fine-tuned or steered model outputs.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses bias, robustness, evaluation.

Abstract

arXiv:2605.11341v1 Announce Type: new Abstract: This paper presents CPEMH, an agentic framework designed to evaluate prompt-driven behavior in foundation-model systems operating on transcript-based datasets for mental-health screening. CPEMH serves as an engineering methodology for behavioral assurance in large-scale language systems, introducing an orchestrated architecture that autonomously performs the design, evaluation, and selection of prompt strategies, enabling systematic control of behavioral variability across contexts. Its modular agentic design, combining orchestrator, inference, and evaluation agents, ensures traceability, reproducibility, and robustness throughout the prompting lifecycle. A case study on automated depression screening from interview transcripts demonstrates the framework's capacity to stabilize and audit foundation-model behavior in conversational and clinically sensitive domains. Lessons learned emphasize the role of modular orchestration in behavioral assurance, the prioritization of stability over architectural complexity, and the integration of F1, bias, and robustness as core acceptance criteria.