feat: add --check-exploits adversarial assertion audit - #41
Merged
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
Add an audit mode that mirrors --check-reference-solution: where that proves a correct solution passes, --check-exploits proves zero-effort cheats fail. For every execution test it synthesizes a battery of trivial stubs from the test_function signature (empty body; return 1/0/true/ false/null/''/array()), runs each through the real verifier, and flags any test whose assertions a cheat can satisfy. A flagged test is under-specified: its assertions check a predictable output (one fixture's expected answer) rather than the WordPress behavior the task describes, so a model could score without doing the work. This is the failure mode that invalidated parts of SWE-bench Verified (insufficient tests) and is exactly the blind spot --check-reference-solution cannot see, since correct code and a lucky constant both pass. - New wp_bench/exploits.py: generic candidate synthesis (extensible with per-test authored exploit_solutions later — read via getattr today). - core.py: _run_exploit_audit / _run_exploit_audit_tests, serial with a reset before every candidate; short-circuits a test on first passing cheat; exits non-zero if any test is exploitable. - config.py: run.check_exploits + validators (exclusive with check_reference_solution and dry_run). - cli.py: --check-exploits flag. output.py: findings table + summary. - Results file records the passing cheat per exploitable test; metadata.audit carries counts and exploitable_test_ids. Verified: ruff clean, mypy python clean, pytest python 142 passed (9 new). Live run against the suite reported separately in the PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Behavior-preserving cleanup from the four-angle review: - Dedup: delete _ensure_reference_solution_ids_are_execution_tests; both modes use the general _ensure_execution_only_ids. - Dedup: _build_verification_spec now parses the gateway name via exploits.gateway_name() instead of a second copy of the regex, so the audit and the verifier agree on what the gateway is. Drops import re. - Altitude: the audit record is built through a new records.build_exploit_audit_record() (shared header centralized, per the records.py contract) instead of a hand-rolled dict. - Simplify: drop the redundant `auditable` record field (it equals candidates_tried > 0); derive where consumed. - Simplify: extract _first_passing_exploit(); flattens the candidate loop and hoists _build_verification_spec to once per test (was rebuilt per candidate). - Simplify/efficiency: the exploitable/auditable partition is computed once in the runner and passed to print_exploit_findings() rather than recomputed there. - Efficiency: order the generic battery constants-first (empty-stub last) so an exploitable test reports the more informative cheat and stops sooner. Skipped (noted): a shared _base_metadata() helper across run() / MultiModelRunner / the audit — that refactors pre-existing code outside this change. Kept the generic/authored exploit_candidates seam (the follow-up that adds the exploit_solutions schema field will use it). Verified: ruff clean, mypy python clean, pytest python 142 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lezama
force-pushed
the
feat/exploit-audit
branch
from
July 17, 2026 14:10
a60605c to
61676cb
Compare
lezama
added a commit
that referenced
this pull request
Jul 17, 2026
…its (#43) ## What Hardens the 8 execution tests that the #41 exploit audit (`--check-exploits`) flagged as exploitable: a zero-effort constant return (`return true;`, `return 1;`, `return false;`) satisfied their runtime assertions. All 8 shared the same root cause: the assertion graded the function's return value against a single fixed expectation instead of observing WordPress state, so it could not tell a real implementation from a hard-coded answer. ## How each test is hardened Every assertion now verifies behavior a constant cannot fake: | Test | Old weakness | New verification | |---|---|---| | `e-ai-client-001` | `return false` passed | Counting spy on the `wp_supports_ai` filter proves the support state is actually read during the call, plus the existing re-enabled-after check | | `e-ai-client-005` | `return false` passed | Late spy (`PHP_INT_MAX`) on `wp_ai_client_prevent_prompt` proves prevention is observed **active during** the support check, and inactive after | | `e-comments-users-004` | `return 1` passed | Second fixture post with zero comments; expected counts become `{2, 0}`, so no single constant passes both calls | | `e-connectors-api-001` | `return true` passed | Helper is called twice: with Akismet registered (expects `true`) and temporarily unregistered via `WP_Connector_Registry::unregister()` (expects `false`), then restored | | `e-post-types-taxonomy-001` | assertion returned the function's own `$ok` | Inspects `get_post_type_object( 'wpbp_book' )`: exists, `public`, `show_in_rest` | | `e-post-types-taxonomy-002` | same | Inspects `get_taxonomy( 'wpbp_genre' )`: exists, `hierarchical`, attached to `post` | | `e-post-types-taxonomy-003` | assertion was literally `return f();` | Fixtures (`wpbp_movie`, `wpbp_topic`) move to runtime `setup`/`teardown`; assertion checks the actual attachment with `is_object_in_taxonomy()` | | `e-roles-caps-002` | `return true` passed | Spy on `updated_option` for the roles option observes `wpbp_temp_cap` actually being **added** to Subscriber before the final not-present check | ## Design decisions - **Slugs moved into prompts.** Models only see `prompt` + `requirements` + `test_function` (never `expected_behavior`), so state-based assertions on `wpbp_book` / `wpbp_genre` / `wpbp_topic` / `wpbp_movie` require the prompts to name them. Done for the three post-types tests. - **Dropped the "Clean up registered types when possible" requirement** on `e-post-types-taxonomy-001/002`. A post-call state inspection requires the registration to still exist when the assertion runs; cleanup is now the assertion's job (as the reference solutions already assumed). - **`e-connectors-api-001` keeps its contract** (no-arg helper, `akismet` required pattern, `WP_Connector_Registry` forbidden for the model). The registry-toggle happens in assertion code, which static checks don't apply to. A lazy-init spy on `wp_connectors_init` was tried first but the registry initializes during `init`, before assertions run. - **`e-connectors-api-002`-style negative cases** were preferred over spies wherever the API allowed it; spies are only used where the observable is a filter/option side effect (`ai-client`, `roles-caps`). - Each touched suite file bumps `2.1.0` → `2.2.0`, same convention as #24. ## Verification (live WP 7.0 runtime) Both directions of the audit loop, against the real wp-env grader: - `--check-reference-solution` on the 8 tests: **8/8 pass** (correctness 1.0) - `--check-exploits` on the 8 tests: **0/8 exploitable** - Full pytest suite: 151 passed; dataset validation green - `--check-exploits` on the full suite: **0/140 auditable tests exploitable** (was 8/140 at the #41 baseline). Full end-to-end run: 1120 exploit-candidate executions, each against a freshly reset WordPress, ~1.5h wall clock. The 10 non-auditable tests have no `test_function` or plugin artifact for the generic battery to target, unchanged from the baseline. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new audit mode that is the mirror image of
--check-reference-solution:--check-reference-solution(existing)--check-exploits(this PR)For every execution test,
--check-exploitssynthesizes a battery of trivial stubs from thetest_functionsignature —return 1 / 0 / true / false / null / '' / array()and an empty body — runs each through the real WordPress verifier, and flags any test whose assertions a cheat can satisfy. It exits non-zero if any test is exploitable and writes the passing cheat per test to the results file (metadata.audit.exploitable_test_ids).Why — and what the live run found
A flagged test is under-specified: its assertions check a predictable output — one fixture's expected answer — rather than the WordPress behavior the task describes. A model can then score without doing the work. This is exactly the failure mode that invalidated parts of SWE-bench Verified (insufficient tests; fixing them reordered 24% of agents in the UTBoost audit), and it is the precise blind spot
--check-reference-solutioncannot see: correct code and a lucky constant both pass.Live run against the full suite (WordPress 7.0 runtime): 8 of 140 auditable tests are exploitable by a zero-effort cheat (10 of 150 tests have no
test_function/are plugin artifacts, so the generic battery doesn't cover them):e-ai-client-001return false;e-ai-client-005return false;e-comments-users-004return 1;e-connectors-api-001return true;e-post-types-taxonomy-001return 1;e-post-types-taxonomy-002return 1;e-post-types-taxonomy-003return 1;e-roles-caps-002return 1;Every one is a single-fixture assertion whose expected answer is a constant. Example (
e-comments-users-004, "return the number of approved comments"): the setup makes one post with one approved comment and the assertion checks=== 1, sofunction wpbp_comments_users_004() { return 1; }— which touches no WordPress API — scoresexecution_pass=True, identical to the real solution on the primary metric. Hardening the assertion to a multi-fixture, unpredictable count makes the cheat fail while the reference still passes (verified live).Design notes (owning the calls, for review)
exploit_solutionsfor cheats the battery can't express;exploit_candidatesalready composes generic + authored, but the schema field is a deliberate follow-up to keep this PR off the dataset loader/export surface.--check-reference-solution, it needs the live runtime, so it belongs in the release checklist / manual audit, not the Docker-less CI. Exits non-zero on findings so it can gate there.Verification
ruff check python— cleanmypy python— clean (0 errors)pytest python— 142 passed (9 new)/simplifypass (dedup ensure-ids helper + gateway-name regex; audit record through a records.py builder; partition computed once; battery constants-first) — see second commitScore impact
🤖 Generated with Claude Code