Describe the bug

So I’ve been running vally evaluations for the eventhubs tools and I’m starting to see some consistent results…

   - create-consumer-group          NAMESPACE+CONSOLIDATED BOTH VALUABLE - namespace vs consolidated - namespace less efficient than consolidated (worse on 4/4 metrics); tokens 405175 vs 104473 (+300702 (+288%)), turns 16 vs 4 (+12 (+300%)), wall 75.4s vs 25.1s (+50.3s (+200%)), credits 37.11 vs 18.31 (+18.8 (+103%))
    - update-consumer-group          NAMESPACE+CONSOLIDATED BOTH VALUABLE - namespace vs consolidated - namespace less efficient than consolidated (worse on 4/4 metrics); tokens 256985 vs 104522 (+152463 (+146%)), turns 10 vs 4 (+6 (+150%)), wall 50.5s vs 25.5s (+25.0s (+98%)), credits 27.41 vs 18.30 (+9.1 (+50%))

Note the comment about namespace less efficient. This is consistent across all the experiments I’ve been running for eventhubs – consolidated mode is dramatically better than namespace mode.

I dug into the logs and it turns out that the root cause of this is that at the start of the prompt evaluation, the LLM calls into either the azure-eventhubs tool (namespace mode) or the azure-get_azure_messaging_service_details tool (consolidated mode).

And the response for each is dramatically different. The azure-eventhubs tool returns a 23K json blob and the azure-get_azure_messaging_service_details tool returns a 7K json blob.

The problem is that copilot sees the 23K blob and decides it’s too long and spills the file over into a file in the session state directory. The LLM then proceeds to spend a significant number of turns trying to find the file and read its contents (in the examples above 6-12 turns).

And that explains pretty much the entire difference between the two MCP scenarios. Basically the extra large response from the azure-eventhubs tool breaks the LLM.

Jason and I spent some time digging into why namespace mode is so large and the core difference is that the azure-eventhubs response contains all the tools in the eventhubs namespace, but the azure-get_azure_messaging_service_details tool returns a much smaller set of tools.

Specifically, the azure-eventhubs tool returns the following tools when prompted with “Update a consumer group's user metadata in an event hub”:
Eventhubs_eventhub_consumergroup_delete
Eventhubs_eventhub_consumergroup_get
Eventhubs_eventhub_consumergroup_update
Eventhubs_eventhub_delete
Eventhubs_eventhub_get
Eventhubs_eventhub_update
Eventhubs_namespace_delete
Eventhubs_namespace_update
Eventhubs_namespace_get

The azure-get_azure_messaging_service_details tool returns the following set of tools for the prompt “Update consumer group in event hub”:
Eventhubs_eventhub_namespace_get
Eventhubs_eventhub_eventhub_consumergroup_get
Eventhubs_eventhub_get

Note that the consolidated mode list also contains the following tools which are NOT returned:
"eventgrid_topic_list",
"eventgrid_subscription_list",
"servicebus_queue_details",
"servicebus_topic_details",
"servicebus_topic_subscription_details",
"eventhubs_namespace_get",
"eventhubs_eventhub_get",
"eventhubs_eventhub_consumergroup_get"

So the consolidated mode tool is filtering the response as well.

It’s clear from the evaluations that it’s important to reduce the size of the tool list returned from learn mode to be below the 20K or so limit on the size of a response.

I think splitting the eventhubs mode into azure-eventhubs_nondestructive and azure-eventhubs_destructive might be sufficient to solve this particular problem but this is a general problem that we can probably build tests for.

Expected behavior

Namespace mode and consolidated modes cost should be similar.

Actual behavior

namespace is WAY worse than consolidated mode from acost basis.

Reproduction Steps

Run a vally experiment contrasting eventhubs use for consolidated vs namespace mode.

Environment

No response