Skip to content

fix: harden the 8 assertion sets flagged exploitable by --check-exploits - #43

Merged
lezama merged 1 commit into
trunkfrom
fix/harden-exploitable-assertions
Jul 17, 2026
Merged

fix: harden the 8 assertion sets flagged exploitable by --check-exploits#43
lezama merged 1 commit into
trunkfrom
fix/harden-exploitable-assertions

Conversation

@lezama

@lezama lezama commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What

Hardens the 8 execution tests that the #41 exploit audit (--check-exploits) flagged as exploitable: a zero-effort constant return (return true;, return 1;, return false;) satisfied their runtime assertions.

All 8 shared the same root cause: the assertion graded the function's return value against a single fixed expectation instead of observing WordPress state, so it could not tell a real implementation from a hard-coded answer.

How each test is hardened

Every assertion now verifies behavior a constant cannot fake:

Test Old weakness New verification
e-ai-client-001 return false passed Counting spy on the wp_supports_ai filter proves the support state is actually read during the call, plus the existing re-enabled-after check
e-ai-client-005 return false passed Late spy (PHP_INT_MAX) on wp_ai_client_prevent_prompt proves prevention is observed active during the support check, and inactive after
e-comments-users-004 return 1 passed Second fixture post with zero comments; expected counts become {2, 0}, so no single constant passes both calls
e-connectors-api-001 return true passed Helper is called twice: with Akismet registered (expects true) and temporarily unregistered via WP_Connector_Registry::unregister() (expects false), then restored
e-post-types-taxonomy-001 assertion returned the function's own $ok Inspects get_post_type_object( 'wpbp_book' ): exists, public, show_in_rest
e-post-types-taxonomy-002 same Inspects get_taxonomy( 'wpbp_genre' ): exists, hierarchical, attached to post
e-post-types-taxonomy-003 assertion was literally return f(); Fixtures (wpbp_movie, wpbp_topic) move to runtime setup/teardown; assertion checks the actual attachment with is_object_in_taxonomy()
e-roles-caps-002 return true passed Spy on updated_option for the roles option observes wpbp_temp_cap actually being added to Subscriber before the final not-present check

Design decisions

  • Slugs moved into prompts. Models only see prompt + requirements + test_function (never expected_behavior), so state-based assertions on wpbp_book / wpbp_genre / wpbp_topic / wpbp_movie require the prompts to name them. Done for the three post-types tests.
  • Dropped the "Clean up registered types when possible" requirement on e-post-types-taxonomy-001/002. A post-call state inspection requires the registration to still exist when the assertion runs; cleanup is now the assertion's job (as the reference solutions already assumed).
  • e-connectors-api-001 keeps its contract (no-arg helper, akismet required pattern, WP_Connector_Registry forbidden for the model). The registry-toggle happens in assertion code, which static checks don't apply to. A lazy-init spy on wp_connectors_init was tried first but the registry initializes during init, before assertions run.
  • e-connectors-api-002-style negative cases were preferred over spies wherever the API allowed it; spies are only used where the observable is a filter/option side effect (ai-client, roles-caps).
  • Each touched suite file bumps 2.1.02.2.0, same convention as Add test_function to execution tests #24.

Verification (live WP 7.0 runtime)

Both directions of the audit loop, against the real wp-env grader:

  • --check-reference-solution on the 8 tests: 8/8 pass (correctness 1.0)

  • --check-exploits on the 8 tests: 0/8 exploitable

  • Full pytest suite: 151 passed; dataset validation green

  • --check-exploits on the full suite: 0/140 auditable tests exploitable (was 8/140 at the feat: add --check-exploits adversarial assertion audit #41 baseline). Full end-to-end run: 1120 exploit-candidate executions, each against a freshly reset WordPress, ~1.5h wall clock. The 10 non-auditable tests have no test_function or plugin artifact for the generic battery to target, unchanged from the baseline.

🤖 Generated with Claude Code

The #41 exploit audit found 8/140 execution tests where a zero-effort
constant return satisfies the runtime assertions. Every one graded the
function's return value against a single fixed expectation instead of
observing WordPress state, so the assertions could not tell a real
implementation from a hard-coded answer.

Each assertion now verifies behavior a constant cannot fake:

- e-ai-client-001: a counting spy on the wp_supports_ai filter proves
  the support state is actually read during the call.
- e-ai-client-005: a late spy on wp_ai_client_prevent_prompt proves
  prevention is observed active during the support check.
- e-comments-users-004: a second fixture post with zero comments makes
  the expected counts {2, 0}, so no single constant passes.
- e-connectors-api-001: the helper is called twice, with Akismet
  registered and temporarily unregistered, and must reflect live
  registry state (restored afterward).
- e-post-types-taxonomy-001/002: assertions inspect the registered
  post type/taxonomy object (public, show_in_rest, hierarchical,
  attached to post) instead of trusting the function's return.
- e-post-types-taxonomy-003: fixtures move to runtime setup/teardown
  and the assertion checks is_object_in_taxonomy() for the actual
  attachment.
- e-roles-caps-002: a spy on updated_option observes wpbp_temp_cap
  actually being added to the Subscriber role before its removal.

Slugs the assertions now depend on (wpbp_book, wpbp_genre, wpbp_topic,
wpbp_movie) are stated in the prompts, since models never see
expected_behavior. Each touched suite file bumps to 2.2.0.

Live-verified against the WP 7.0 runtime in both directions:
--check-reference-solution passes 8/8 and --check-exploits reports
0/8 exploitable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: lezama <migueluy@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@lezama

lezama commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

full suite re-audit finished clean: 0/140 auditable tests exploitable (1120 exploit executions against the live WP 7.0 runtime, each on a fresh reset). updated the PR body with the final numbers 🎉

@lezama
lezama merged commit 1e5c0f7 into trunk Jul 17, 2026
5 checks passed
@lezama
lezama deleted the fix/harden-exploitable-assertions branch July 17, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant