Skip to content

Add resolved model for events#4210

Merged
lramos15 merged 5 commits intomicrosoft:mainfrom
dennyac:add-resolved-model-for-events
Mar 11, 2026
Merged

Add resolved model for events#4210
lramos15 merged 5 commits intomicrosoft:mainfrom
dennyac:add-resolved-model-for-events

Conversation

@dennyac
Copy link
Copy Markdown
Contributor

@dennyac dennyac commented Mar 4, 2026

Add resolved model to user action telemetry

  • When modelId is copilot/auto, replace it with the actual model that served the request
  • Leverages existing resolvedModel field from the server response, already plumbed in Add resolved model #1517
  • Surfaces resolvedModel in IResultMetadata so downstream consumers can access it
  • Affects events: conversation.acceptedCopy, conversation.acceptedInsert, conversation.appliedCodeblock
  • Non-auto model selections are unchanged
Copilot AI review requested due to automatic review settings March 4, 2026 22:09
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for recording the resolved model (the actual model that served a request) in user-action telemetry, especially when the user selected copilot/auto, by propagating the server-provided resolvedModel through result metadata and using it in the affected telemetry events.

Changes:

  • Plumbs resolvedModel into IResultMetadata and populates it from successful ChatResponse results.
  • Updates user action telemetry to substitute copilot/auto with the resolved model for copy/insert/apply codeblock events.
  • Updates the static chat fetcher test helper to allow configuring resolvedModel on responses; adds a new test file.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/platform/chat/test/common/staticChatMLFetcher.ts Allows test fetcher responses to return a configurable resolvedModel.
src/extension/prompt/node/defaultIntentRequestHandler.ts Copies response.resolvedModel into the metadata fragment on successful responses.
src/extension/prompt/common/conversation.ts Extends IResultMetadata with optional resolvedModel.
src/extension/conversation/vscode-node/userActions.ts Replaces telemetry modelId with metadata.resolvedModel when the action model is copilot/auto.
src/extension/conversation/vscode-node/test/userActionsResolvedModel.spec.ts Adds tests, but currently re-implements the production mapping logic instead of exercising it.
Comments suppressed due to low confidence (1)

src/extension/conversation/vscode-node/userActions.ts:281

  • The copilot/auto resolution logic and fallback string are duplicated inline here and again in handleApplyAction. To reduce the chance of future drift, consider extracting a small helper (and a shared AUTO_MODEL_ID constant) that maps (requestedModelId, resolvedModel) to the telemetry modelId.
					languageId: e.action.languageId ?? '',
					modelId: e.action.modelId === 'copilot/auto' ? (result.metadata?.resolvedModel || 'copilot/auto') : (e.action.modelId ?? ''),
					comp_type: compType,
Comment on lines 149 to 153
const metadataFragment: Partial<IResultMetadata> = {
toolCallRounds: resultDetails.toolCallRounds,
toolCallResults: this._collectRelevantToolCallResults(resultDetails.toolCallRounds, resultDetails.toolCallResults),
resolvedModel: resultDetails.response.type === ChatFetchResponseType.Success ? resultDetails.response.resolvedModel : undefined,
};
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There’s no unit test asserting that resolvedModel is actually propagated into chatResult.metadata from a successful ChatResponse. Since defaultIntentRequestHandler.spec.ts already covers getResult(), it would be better to add an assertion there (e.g., set the test IChatMLFetcher to return a non-empty resolvedModel and verify it appears in the returned metadata).

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@dennyac dennyac force-pushed the add-resolved-model-for-events branch from 256ab5b to efde35c Compare March 5, 2026 00:59
@dennyac
Copy link
Copy Markdown
Contributor Author

dennyac commented Mar 5, 2026

Tagging @lramos15 as it's related to #1517

@aiday-mar aiday-mar requested a review from ulugbekna March 5, 2026 09:26
@aiday-mar aiday-mar removed their assignment Mar 5, 2026
@lramos15 lramos15 self-assigned this Mar 5, 2026
@lramos15 lramos15 enabled auto-merge March 11, 2026 19:30
@vs-code-engineering vs-code-engineering bot added this to the 1.112.0 milestone Mar 11, 2026
@lramos15 lramos15 added this pull request to the merge queue Mar 11, 2026
Merged via the queue into microsoft:main with commit 9b79523 Mar 11, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

5 participants