api_dispatch silently converts a text-block-free API response into an empty-string SUCCESS (7/10 Fable calls lost in #2202)
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
The bug
src/explore_persona_space/llm/api_dispatch.py:820 (and the identical line at
:1163):
text = next((b.text for b in msg.content if b.type == "text"), "")
When an API response carries no text content block — the model emitted
only a thinking block, the output budget was exhausted before any text was
produced, the response was a refusal with no text, or the content list is empty
— this defaults to the empty string and the item is recorded as a success:
{"result": "", "error": false, "reason": null, "category": "ok"}
No exception, no error flag, no reason, category: "ok". The caller cannot
distinguish "the model returned nothing" from "the model returned an empty
answer on purpose". This is a silent default swallowing a fault — the exact
shape CLAUDE.md's fail-fast rule forbids ("no value placeholders, silent
defaults, or fallbacks that swallow the fault; the crash IS the signal").
This is a shared-library path: every caller of dispatch_calls inherits it,
not just the one that found it.
How it was found (#2202)
In #2202, 7 of 10 Fable-5 digest chunks returned empty. The failure digest was
5/5 empty; the 500-sample digest was 2 complete, 1 truncated mid-word, 2 empty.
Every one of the seven was classified category: "ok". Downstream,
parse_modes("") returned the empty list, so the empties contributed zero
proposals and the pipeline continued as though those chunks had merely found
nothing — mode discovery silently ran on 30% of its intended input and the run
reached awaiting_promotion with the gap visible only as a line in the
coverage-notes takeaway.
Aggravating factor at the call site: the caller passed
parse_response=lambda t: t rather than parse_response_meta, so stop_reason
was never persisted and the PROXIMATE cause is now unrecoverable from the
artifacts. That is a caller-side choice, but the library makes the safe choice
easy to skip: the metadata-aware parser is opt-in while the lossy default is
free.
Why the existing gates did not catch it
- The llm-judging rule-26 pilot gate keys on
stop_reason == "max_tokens", which requiresparse_response_meta. With the plain parser there is no field to gate on. - No gate anywhere treats an empty model reply as a failure condition.
- The #2202 pilot probe was a 16-token
"Reply with the single word OK."round-trip, which exercises auth and routing but not the production prompt shape (the sibling-axis version of #2152 — a pilot must exercise the instrument it gates).
Proposed fix
- Make the empty case explicit at the library boundary. When no
textblock is present, do NOT return the empty string as a success. Either raise, or return a typed failure record carryingcategory(for exampleempty_response), the responsestop_reason, and the content-block types actually returned, so the caller sees WHY. Preserve the existingerror/reason/categoryrecord contract so current consumers keep working. - Persist
stop_reasonon every path, not only when a caller opts intoparse_response_meta— the field is already getattr-read at:823/:1167; record it on the result object regardless of which parser the caller passed. - Audit and migrate existing callers. Enumerate every
dispatch_callscall site; any that can be fed a non-trivial prompt should either handle the typed empty record or opt into the metadata parser. Report the list. - Add a regression test that a mocked response with no text block (and one
with an empty content list) produces a FAILURE record, not
{"result": "", "category": "ok"}— plus a test thatstop_reasonsurvives on the default parser path. - Consider a lint check in
workflow_lint.pyfor thenext((... type == "text" ...), "")shape, so the pattern cannot silently reappear elsewhere.
Acceptance criteria
- A text-block-free response is a typed failure, never a success with an empty
result, at both
:820and:1163. stop_reasonis persisted on every dispatch path.- Every existing
dispatch_callscaller is enumerated and either migrated or explicitly recorded as unaffected. - Regression tests cover the no-text-block and empty-content-list cases.
- The full relevant test suite is green on
main.
Files of record
src/explore_persona_space/llm/api_dispatch.py:820,:1163,:823,:1167- #2202 (discovery;
eval_results/issue_2202/fable_reads/, dispatcher cachedata/issue_2202/fable_cache/),scripts/issue2202_labels.py:409(the caller-side check that an empty string passes) - #2152 (pilot gate must exercise the instrument it gates — sibling, not a duplicate: that one is about the pilot's transport, this one is about the library's silent default)
.claude/rules/llm-judging.mdrules 23/26 (max_tokens + pilot gating)