Skip to content

feat: add run.continue_on_error to record per-test failures and keep going - #40

Merged
lezama merged 2 commits into
trunkfrom
feat/continue-on-error
Jul 17, 2026
Merged

feat: add run.continue_on_error to record per-test failures and keep going#40
lezama merged 2 commits into
trunkfrom
feat/continue-on-error

Conversation

@lezama

@lezama lezama commented Jul 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

New run.continue_on_error switch (default false): when enabled, an unrecoverable per-test error — a provider failure that survives the retry policy, or an unexpected harness exception — is warned about on one line, recorded as a canonical result record with the error field populated, and the run continues to the next test.

  • Errored records fill the schema's reserved seam: error: {type, message} (reserved null since the unified schema landed), null scores, exact canonical key structure — no RESULT_SCHEMA_VERSION bump, no fourth parser needed.
  • Excluded from aggregates, by design: an ungraded test neither passes nor fails, so infrastructure failures never masquerade as model failures in execution_pass_rate/knowledge. The trade-off (a diagnostic run's denominator shrinks) is disclosed in the results file via metadata.continue_on_error and metadata.errored_test_ids — an auditor can always see exactly which tests were skipped and why the run is not leaderboard material.
  • Systemic guardrail, two layers: (1) if the first 5 processed tests all error with zero successes, abort immediately — bad credentials or a dead runtime should not burn 470 tests of wall-clock and spend; (2) a run that completes with errors and zero graded tests (e.g. --limit 3, all erroring) also aborts instead of writing a useless null-score results file. One graded success disarms both — after that, errors are per-test facts worth recording.
  • Applies to knowledge, execution, and reference-solution paths (an errored reference test shows up in the failure table → exit 1, so authors see all broken tests in one pass instead of dying at the first).
  • Incidental de-dup: the two identical knowledge futures loops (single/multi-model) collapse into a shared _run_knowledge_loop(), mirroring _run_isolated_execution_loop.

Why

A full run is hours of compute and thousands of model calls. The retry layer (#32) absorbs transient blips, but a single error that survives it currently aborts the entire run and discards everything already graded. For long diagnostic runs the right cost of one bad test is one record — while official leaderboard runs keep the strict default: every selected test must grade, so the flag is off by default and documented as diagnostic-only (same contract as skip_runtime/execution_isolation: none).

This is deliberately the first slice of the "distinguish infrastructure failures from model failures" direction noted in #32 — resumable runs can build on the same error-record shape later.

Verification

  • ruff check python — clean
  • mypy python — clean (0 errors)
  • pytest python — 142 passed (9 new: default-off still aborts; error recorded + run continues on serial, concurrent, and knowledge paths; aggregates exclude errored tests; metadata audit fields; systemic abort after 5 straight errors; all-error run below the threshold aborts too; one success disarms the guardrail)
  • PHP syntax / runtime smoke test — n/a (runtime/ untouched)

Score impact

  • This PR does not change how any test is scored. Default behavior is byte-identical (flag off → legacy abort). With the flag on, graded tests score exactly as before; errored tests carry null scores and are excluded from aggregates — and such runs are diagnostic-only by documentation, not leaderboard artifacts.

🤖 Generated with Claude Code
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: lezama <migueluy@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.
@lezama @claude
feat: add run.continue_on_error to record per-test failures and keep …
1d942b7 
…going

A single unrecoverable per-test error (provider failure surviving
retries, unexpected harness exception) currently aborts the whole run,
discarding hours of compute and API spend. For long diagnostic runs
that trade-off is wrong: one bad test should cost one record, not the
run.

- New run.continue_on_error (default false — legacy abort semantics,
  required for official runs). When enabled, a per-test TestError is
  warned about on one line, recorded, and the run continues.
- Errored tests become canonical records via build_error_record(),
  filling the error field reserved by the unified schema: null scores
  (an ungraded test neither passes nor fails), exact canonical key
  structure, no schema bump.
- Errored tests are excluded from score aggregates so infrastructure
  failures never masquerade as model failures; they are auditable via
  metadata.continue_on_error and metadata.errored_test_ids.
- Systemic guardrail: if the first 5 processed tests all error with no
  success, the run aborts anyway with a clear message — a bad API key
  or dead runtime should fail loudly, not burn the suite.
- Knowledge loops (single- and multi-model) deduplicated into a shared
  _run_knowledge_loop(), mirroring _run_isolated_execution_loop.
- Applies to knowledge, execution, and reference-solution paths; the
  reference failure table already surfaces errored records as failures.

Verified: pytest python (141 passed, 8 new), ruff clean, mypy python
clean (0 errors).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lezama
lezama force-pushed the feat/continue-on-error branch from 212710e to 1d942b7 Compare July 10, 2026 20:24
@lezama @claude
refactor: address /simplify review of continue_on_error
cc28ca2 
Behavior-preserving cleanup from the four-angle review:

- Extract _run_concurrent_loop(): the new _run_knowledge_loop was
  byte-for-byte the concurrent branch of _run_isolated_execution_loop
  (only worker count + label differed). Both now route through one helper;
  the serial reset_per_test branch — the genuinely different loop — stays.
- Extract records._base_record(): the three record builders each re-listed
  the full canonical key skeleton (the keys-equality test existed to police
  that triplication). They now override a shared skeleton, so the shape
  can't drift.
- Add core._error_record(): the TestError -> record extraction
  (type/message/test_type) was copy-pasted into all five on_error closures;
  each is now a one-liner.
- Add records.errored_test_ids(): the same comprehension appeared in two
  metadata blocks.
- Rename _ContinueOnErrorPolicy.should_abort -> register_error: it is a
  mutating predicate, so the name now signals the side effect. Documented
  that the policy's per-loop scope is intentional.

Skipped (noted): giving ScoreAggregator a record-aware add_record() to own
the "errored record contributes nothing" guard — it would couple the
aggregator to the record schema (reaches into scoring.py, outside this
change), and the reviewers split on it.

Verified: ruff clean, mypy python clean, pytest python 142 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lezama
lezama merged commit bd5321b into trunk Jul 17, 2026
5 checks passed
@lezama
lezama deleted the feat/continue-on-error branch July 17, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant

@lezama