Skip to content

fix(scan): name garak check results after probe detector plugins - #2585

Merged
henchaves merged 1 commit into
feat/garak-scan-integrationfrom
cursor/garak-check-names-f3f6
Jul 6, 2026
Merged

fix(scan): name garak check results after probe detector plugins#2585
henchaves merged 1 commit into
feat/garak-scan-integrationfrom
cursor/garak-check-names-f3f6

Conversation

@kevinmessiaen

Copy link
Copy Markdown
Member

Summary

Garak scan results were showing generic check_1, check_2, … labels because CheckResult details lacked check_name. Giskard uses check_name for display and JUnit export.

Garak itself keys detector results by the short plugin name declared on the probe (e.g. always.Fail, judge.Refusal), matching garak.harnesses.base behavior. This change propagates that label into every garak-derived CheckResult.

Changes

  • _resolve_detectors now returns (label, detector) pairs using the probe's primary_detector / extended_detectors names
  • _evaluate_attempt sets check_name and detector details from that label (including judge wrappers where detectorname would otherwise be a dynamic class path)
  • Metric names use the same detector label instead of detector.name (which often falls back to the base Detector class)
  • Tests updated to assert proper check_name values

Verification

make format
make test-garak   # 38 passed
make test-unit PACKAGE=giskard-scan  # 187 passed, 4 deselected
Open in Web Open in Cursor 
Garak keys detector results by the short plugin name from the probe
configuration (e.g. always.Fail, judge.Refusal), not detector.name.
Populate check_name and detector details with that label so scan output
no longer shows generic check_1, check_2 names.

Co-authored-by: Kevin Messiaen <kevinmessiaen@users.noreply.github.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Garak scan adapter to return and track detector labels alongside detector instances. Specifically, _resolve_detectors now returns a list of (detector_label, detector) tuples, allowing the adapter to correctly identify and report the detector name as check_name in the results. The corresponding tests have been updated to align with this new signature. There are no review comments to address, so I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@kevinmessiaen
kevinmessiaen marked this pull request as ready for review July 3, 2026 13:35
@henchaves
henchaves self-requested a review July 6, 2026 08:02
@henchaves
henchaves merged commit ca247d3 into feat/garak-scan-integration Jul 6, 2026
45 checks passed
@henchaves
henchaves deleted the cursor/garak-check-names-f3f6 branch July 6, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

3 participants