fix: harden the 8 assertion sets flagged exploitable by --check-exploits - #43
Merged
Conversation
The #41 exploit audit found 8/140 execution tests where a zero-effort constant return satisfies the runtime assertions. Every one graded the function's return value against a single fixed expectation instead of observing WordPress state, so the assertions could not tell a real implementation from a hard-coded answer. Each assertion now verifies behavior a constant cannot fake: - e-ai-client-001: a counting spy on the wp_supports_ai filter proves the support state is actually read during the call. - e-ai-client-005: a late spy on wp_ai_client_prevent_prompt proves prevention is observed active during the support check. - e-comments-users-004: a second fixture post with zero comments makes the expected counts {2, 0}, so no single constant passes. - e-connectors-api-001: the helper is called twice, with Akismet registered and temporarily unregistered, and must reflect live registry state (restored afterward). - e-post-types-taxonomy-001/002: assertions inspect the registered post type/taxonomy object (public, show_in_rest, hierarchical, attached to post) instead of trusting the function's return. - e-post-types-taxonomy-003: fixtures move to runtime setup/teardown and the assertion checks is_object_in_taxonomy() for the actual attachment. - e-roles-caps-002: a spy on updated_option observes wpbp_temp_cap actually being added to the Subscriber role before its removal. Slugs the assertions now depend on (wpbp_book, wpbp_genre, wpbp_topic, wpbp_movie) are stated in the prompts, since models never see expected_behavior. Each touched suite file bumps to 2.2.0. Live-verified against the WP 7.0 runtime in both directions: --check-reference-solution passes 8/8 and --check-exploits reports 0/8 exploitable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
Contributor
Author
|
full suite re-audit finished clean: 0/140 auditable tests exploitable (1120 exploit executions against the live WP 7.0 runtime, each on a fresh reset). updated the PR body with the final numbers 🎉 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Hardens the 8 execution tests that the #41 exploit audit (
--check-exploits) flagged as exploitable: a zero-effort constant return (return true;,return 1;,return false;) satisfied their runtime assertions.All 8 shared the same root cause: the assertion graded the function's return value against a single fixed expectation instead of observing WordPress state, so it could not tell a real implementation from a hard-coded answer.
How each test is hardened
Every assertion now verifies behavior a constant cannot fake:
e-ai-client-001return falsepassedwp_supports_aifilter proves the support state is actually read during the call, plus the existing re-enabled-after checke-ai-client-005return falsepassedPHP_INT_MAX) onwp_ai_client_prevent_promptproves prevention is observed active during the support check, and inactive aftere-comments-users-004return 1passed{2, 0}, so no single constant passes both callse-connectors-api-001return truepassedtrue) and temporarily unregistered viaWP_Connector_Registry::unregister()(expectsfalse), then restorede-post-types-taxonomy-001$okget_post_type_object( 'wpbp_book' ): exists,public,show_in_reste-post-types-taxonomy-002get_taxonomy( 'wpbp_genre' ): exists,hierarchical, attached toposte-post-types-taxonomy-003return f();wpbp_movie,wpbp_topic) move to runtimesetup/teardown; assertion checks the actual attachment withis_object_in_taxonomy()e-roles-caps-002return truepassedupdated_optionfor the roles option observeswpbp_temp_capactually being added to Subscriber before the final not-present checkDesign decisions
prompt+requirements+test_function(neverexpected_behavior), so state-based assertions onwpbp_book/wpbp_genre/wpbp_topic/wpbp_movierequire the prompts to name them. Done for the three post-types tests.e-post-types-taxonomy-001/002. A post-call state inspection requires the registration to still exist when the assertion runs; cleanup is now the assertion's job (as the reference solutions already assumed).e-connectors-api-001keeps its contract (no-arg helper,akismetrequired pattern,WP_Connector_Registryforbidden for the model). The registry-toggle happens in assertion code, which static checks don't apply to. A lazy-init spy onwp_connectors_initwas tried first but the registry initializes duringinit, before assertions run.e-connectors-api-002-style negative cases were preferred over spies wherever the API allowed it; spies are only used where the observable is a filter/option side effect (ai-client,roles-caps).2.1.0→2.2.0, same convention as Add test_function to execution tests #24.Verification (live WP 7.0 runtime)
Both directions of the audit loop, against the real wp-env grader:
--check-reference-solutionon the 8 tests: 8/8 pass (correctness 1.0)--check-exploitson the 8 tests: 0/8 exploitableFull pytest suite: 151 passed; dataset validation green
--check-exploitson the full suite: 0/140 auditable tests exploitable (was 8/140 at the feat: add --check-exploits adversarial assertion audit #41 baseline). Full end-to-end run: 1120 exploit-candidate executions, each against a freshly reset WordPress, ~1.5h wall clock. The 10 non-auditable tests have notest_functionor plugin artifact for the generic battery to target, unchanged from the baseline.🤖 Generated with Claude Code