SkillGen: Verified Inference-Time Agent Skill Synthesis

Authors: Yuchen Ma, Yue Huang, Han Bao et al.

arXiv · PDF

Summary

SkillGen is a system that automatically extracts reusable "skills" (auditable procedures) from an LLM agent's successful and failed trajectories. Rather than just summarizing what worked, it contrasts successes against failures to identify reliable patterns, common failure modes, and missing steps. It then generates candidate skill descriptions, iteratively refines them, and verifies each skill by running the same instances with and without the skill to measure net improvement (accounting for both fixes and regressions). Experiments show SkillGen consistently improves held-out performance, beats summarization baselines, and produces skills that transfer across models.

Main takeaways:

• Most agent skills are still hand-written; SkillGen synthesizes them automatically from execution traces.
• The system uses contrastive induction: comparing successful and failed trajectories to find reusable success patterns and recurring failure modes.
• Skills are human-readable artifacts that can be inspected before deployment, not opaque learned weights.
• A key novelty is modeling skills as interventions: SkillGen tests the same instances with and without the skill to measure repairs (fixed failures) minus regressions (broken successes).
• Across multiple agents and datasets, SkillGen outperforms existing skill-generation baselines and produces skills that transfer to different base models.

Relevance

Tangentially relevant — SkillGen's contrastive induction over success/failure traces and its verification-by-intervention approach echo my interest in finding explicit representations (prompts, descriptions) corresponding to behavioral changes, though SkillGen targets agent tasks rather than persona/assistant behavior.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses failure, failures.

Abstract

arXiv:2605.10999v1 Announce Type: new Abstract: Skills are a promising way to improve LLM agent capabilities without retraining, while keeping the added procedure reusable and controllable. However, high-quality skills are still largely written by hand. We introduce SkillGen, a multi-agent framework that synthesizes a single auditable skill from trajectories generated by a base agent. The output is a human-readable artifact that can be inspected before use. Rather than merely summarizing trajectories, SkillGen leverages contrastive induction over both successful and failed trajectories to identify reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures. SkillGen then generates candidate skills and iteratively refines the skill. A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance. Specifically, we compare outcomes on the same instances with and without the skill, so that we account for both repairs (cases where the skill fixes a baseline failure) and regressions (cases where the skill breaks a baseline success). Across a broad range of agents and datasets, SkillGen consistently improves held-out performance, outperforms existing skill-generation baselines, and produces skills that transfer across models.