fix(scan): name garak check results after probe detector plugins - #2585
Conversation
Garak keys detector results by the short plugin name from the probe configuration (e.g. always.Fail, judge.Refusal), not detector.name. Populate check_name and detector details with that label so scan output no longer shows generic check_1, check_2 names. Co-authored-by: Kevin Messiaen <kevinmessiaen@users.noreply.github.com>
There was a problem hiding this comment.
Code Review
This pull request updates the Garak scan adapter to return and track detector labels alongside detector instances. Specifically, _resolve_detectors now returns a list of (detector_label, detector) tuples, allowing the adapter to correctly identify and report the detector name as check_name in the results. The corresponding tests have been updated to align with this new signature. There are no review comments to address, so I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Summary
Garak scan results were showing generic
check_1,check_2, … labels becauseCheckResultdetails lackedcheck_name. Giskard usescheck_namefor display and JUnit export.Garak itself keys detector results by the short plugin name declared on the probe (e.g.
always.Fail,judge.Refusal), matchinggarak.harnesses.basebehavior. This change propagates that label into every garak-derivedCheckResult.Changes
_resolve_detectorsnow returns(label, detector)pairs using the probe'sprimary_detector/extended_detectorsnames_evaluate_attemptsetscheck_nameanddetectordetails from that label (including judge wrappers wheredetectornamewould otherwise be a dynamic class path)detector.name(which often falls back to the baseDetectorclass)check_namevaluesVerification