Add resolved model for events#4210
Conversation
There was a problem hiding this comment.
Pull request overview
Adds support for recording the resolved model (the actual model that served a request) in user-action telemetry, especially when the user selected copilot/auto, by propagating the server-provided resolvedModel through result metadata and using it in the affected telemetry events.
Changes:
- Plumbs
resolvedModelintoIResultMetadataand populates it from successfulChatResponseresults. - Updates user action telemetry to substitute
copilot/autowith the resolved model for copy/insert/apply codeblock events. - Updates the static chat fetcher test helper to allow configuring
resolvedModelon responses; adds a new test file.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/platform/chat/test/common/staticChatMLFetcher.ts | Allows test fetcher responses to return a configurable resolvedModel. |
| src/extension/prompt/node/defaultIntentRequestHandler.ts | Copies response.resolvedModel into the metadata fragment on successful responses. |
| src/extension/prompt/common/conversation.ts | Extends IResultMetadata with optional resolvedModel. |
| src/extension/conversation/vscode-node/userActions.ts | Replaces telemetry modelId with metadata.resolvedModel when the action model is copilot/auto. |
| src/extension/conversation/vscode-node/test/userActionsResolvedModel.spec.ts | Adds tests, but currently re-implements the production mapping logic instead of exercising it. |
Comments suppressed due to low confidence (1)
src/extension/conversation/vscode-node/userActions.ts:281
- The
copilot/autoresolution logic and fallback string are duplicated inline here and again inhandleApplyAction. To reduce the chance of future drift, consider extracting a small helper (and a sharedAUTO_MODEL_IDconstant) that maps(requestedModelId, resolvedModel)to the telemetrymodelId.
languageId: e.action.languageId ?? '',
modelId: e.action.modelId === 'copilot/auto' ? (result.metadata?.resolvedModel || 'copilot/auto') : (e.action.modelId ?? ''),
comp_type: compType,
src/extension/conversation/vscode-node/test/userActionsResolvedModel.spec.ts
Outdated
Show resolved
Hide resolved
| const metadataFragment: Partial<IResultMetadata> = { | ||
| toolCallRounds: resultDetails.toolCallRounds, | ||
| toolCallResults: this._collectRelevantToolCallResults(resultDetails.toolCallRounds, resultDetails.toolCallResults), | ||
| resolvedModel: resultDetails.response.type === ChatFetchResponseType.Success ? resultDetails.response.resolvedModel : undefined, | ||
| }; |
There was a problem hiding this comment.
There’s no unit test asserting that resolvedModel is actually propagated into chatResult.metadata from a successful ChatResponse. Since defaultIntentRequestHandler.spec.ts already covers getResult(), it would be better to add an assertion there (e.g., set the test IChatMLFetcher to return a non-empty resolvedModel and verify it appears in the returned metadata).
256ab5b to
efde35c
Compare
Add resolved model to user action telemetry