Restores the email_intent benchmark to working order - #11
Merged
Conversation
![github-code-quality[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Restores the email_intent benchmark so it can be loaded and executed end-to-end by aligning dataset inputs, tool specs, and SOP output format with the evaluator and the implemented tools.
Changes:
- Replaced stale/incorrect tool specifications with
get_*tool specs intended to matchtools.py. - Updated benchmark metadata and SOP to include
email_bodyas an input and to emit JSON (wrapped in<final_output>tags) as the expected output format. - Restored the full answer-free dataset (
test_set_without_outputs.csv) and updated tools to read it, adding guards for optional columns and duplicate handling.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/amazon_sop_bench/benchmarks/data/email_intent/toolspecs.json | Replaces the tool catalog with get_* tools and schemas used by the benchmark agent. |
| src/amazon_sop_bench/benchmarks/data/email_intent/tools.py | Switches to the answer-free dataset; adds schema/behavior guards and duplicate handling; adds a __main__ manual-test block. |
| src/amazon_sop_bench/benchmarks/data/email_intent/test_set_without_outputs.csv | Restores the full answer-free dataset (expanded from 62 to 186 rows). |
| src/amazon_sop_bench/benchmarks/data/email_intent/sop.txt | Fixes the tool list and updates the expected output format to JSON wrapped in <final_output> tags. |
| src/amazon_sop_bench/benchmarks/data/email_intent/metadata.json | Corrects input_columns to include email_id, email_body, product_id, and marketplace_id. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| "pattern": "^[A-Z]{2}([0-9]{3})?$" | ||
| } | ||
| }, | ||
| "required": ["product_id", "marketplace_id"] |
| 3.1 Product Attribution System (PAS): The hierarchical database structure maintaining product metadata, including pricing, description, and listing status. | ||
| 3.2 Intent Classification Matrix (ICM): A structured framework for categorizing seller communications into predetermined intent categories. | ||
| 3.3 API Integration Points (AIP): Designated endpoints for retrieving product-specific information including get_product_price, get_product_description, and get_product_description. | ||
| 3.3 API Integration Points (AIP): Designated endpoints for retrieving product-specific information including get_product_price, get_product_description, get_product_listing_status, and get_inventory_status. |
Comment on lines
+359
to
+363
| if __name__ == "__main__": | ||
| product_manager = ProductListingManager() | ||
|
|
||
| ######################## Unit tests for API - get_product_price ######################## | ||
| print("=" * 25) |
![Zuhaib [C] Akhtar](https://cdn.statically.io/img/github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
…ketplace_id validation
metadata.json: input_columns listed tool API params (product_id, marketplace_id) instead of agent task inputs, so the agent never received the email_body it must classify. Restored the inputs used in the original experiments (verified against saved run traces): email_id, email_body, product_id, marketplace_id.
toolspecs.json: advertised 5 tools (classifyEmailIntent etc.) from an abandoned alternate design that were never implemented in tools.py, so every tool call raised 'Invalid tool_name'. Restored the original get_* specs matching tools.py exactly.
tools.py: loosened marketplace_id validation in get_inventory_status and get_product_listing_status from ^[A-Z]{2}[0-9]{3}$ to ^[A-Z]{2}([0-9]{3})?$ to match the tool spec and the bare marketplace codes (US) used in the dataset. Verified: bare and full forms accepted, invalid values still rejected.
…ls.py crash paths Follow-up to the merged input_columns/toolspecs fix. Restores the remaining drifted files to the versions used in the original experiments: sop.txt: section 3.3 listed get_product_description twice and omitted get_product_listing_status and get_inventory_status; section 6.1 instructed XML output while the evaluator parses <final_output> JSON. Restored the complete tool list and JSON output format. tools.py: - guards against missing optional columns (projected_stock, restock_recommendation, update_timestamp) that caused KeyError when agents set include_forecasts/include_history, both advertised in the tool specs - reads test_set_without_outputs.csv (answer-free) instead of test_set_with_outputs.csv - handles duplicate product records with a warning instead of an error test_set_without_outputs.csv: was truncated to 62 rows; restored the full 186-row answer-free dataset matching test_set_with_outputs.csv. Verified end-to-end via BenchmarkLoader: 186 tasks load, agent inputs include email_body, include_forecasts and include_history execute without errors.
- CI installed the package with 'pip install -e .' but no dependencies are declared in pyproject/setup.cfg, so pytest failed at collection (exit 4, ModuleNotFoundError). Install requirements.txt in the workflow. This failure pre-dates this PR (main CI is also red). - Remove unused invalid_response assignments in the tools.py manual test block (CodeQL: unused global variable). - Fix the get_product_listing_status example to use marketplace_id US, matching the dataset row for P78X9Y (was US001, which errored). Verified: 9/9 tests pass locally on the CI command; the tools.py __main__ block runs with all valid examples returning data.
The docstring, usage example, and test_config_defaults all state the default region is us-west-2, but the code fell back to us-east-1 when AWS_REGION was unset. CI sets only AWS_DEFAULT_REGION, so the test failed once dependency installation was fixed (previously pytest died at collection before reaching it). Align the code with the documented default.
groverpr
force-pushed
the
fix/email-intent-restore
branch
from
August 18, 2026 00:07
129714f to
7cf7f57
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restores the email_intent benchmark to working order: correct toolspecs for the 4 implemented tools, input_columns including email_body, SOP output format aligned with the evaluator, full 186-row dataset, and tools.py crash guards plus a data-leakage fix (tools now read the answer-free CSV). Brings the public benchmark to parity with the internal source of truth. Resolves #7.
Details:
6 handles duplicate product records with a warning instead of an error