Skip to content

fix: grade None provider completions as empty answers instead of crashing - #44

Merged
lezama merged 1 commit into
trunkfrom
fix/none-completion-text
Jul 20, 2026
Merged

fix: grade None provider completions as empty answers instead of crashing#44
lezama merged 1 commit into
trunkfrom
fix/none-completion-text

Conversation

@lezama

@lezama lezama commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What

First real multi-model pilot run (gemini-2.5-pro + flash on the hardened dataset) crashed with:

File .../wp_bench/utils.py, line 9, in strip_code_fences
    stripped = output.strip()
AttributeError: 'NoneType' object has no attribute 'strip'

litellm can hand back a choice whose message.content is None — providers do this on safety blocks, thinking-only responses, or truncation. ModelGeneration.text carried the None into parse_artifact, killing the entire run on one bad completion.

Fix

Normalize at the single extraction point in ModelInterface.generate_with_metadata: content or "". An empty completion is a model output, not a harness fault, so it should grade as an empty answer (score 0 for that test) and let the run continue. This also composes with run.continue_on_error for the remaining per-test failure modes.

Includes a regression test with a mocked None-content response.

🤖 Generated with Claude Code

…hing

Found by the first live multi-model pilot: gemini-2.5-pro returned a
response whose message content was None (providers do this on safety
blocks, thinking-only responses, or truncation). ModelGeneration.text
carried the None into parse_artifact -> strip_code_fences, which crashed
the whole run with AttributeError.

An empty completion is a model output, not a harness fault: normalize to
"" at the extraction point so the test grades as an empty answer
(score 0) and the run continues.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: lezama <migueluy@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@lezama
lezama merged commit d20c0e7 into trunk Jul 20, 2026
5 checks passed
@lezama
lezama deleted the fix/none-completion-text branch July 20, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant