A/B experiment: close button vs Skip for now on sign-in dialog#295867
Merged
bpasero merged 1 commit intomicrosoft:mainfrom Feb 18, 2026
Merged
A/B experiment: close button vs Skip for now on sign-in dialog#295867bpasero merged 1 commit intomicrosoft:mainfrom
bpasero merged 1 commit intomicrosoft:mainfrom
Conversation
Add experiment 'chatSetupDialogCloseButton' using IWorkbenchAssignmentService: - Treatment (true): show standard X close button, hide Skip for now button - Control (default): show Skip for now button, hide X close button Both variants produce the same 'failedMaybeLater' telemetry on dismiss. Test locally via: experiments.override.chatSetupDialogCloseButton: true
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements an A/B experiment to compare two dialog dismissal patterns in the chat setup flow: a standard X close button (treatment) versus a "Skip for now" text button (control). The experiment uses the IWorkbenchAssignmentService to randomly assign users to either variant, with both producing identical telemetry when dismissed.
Changes:
- Integrated experiment assignment service to conditionally configure dialog close behavior
- Added logic to toggle between close button and "Skip for now" button based on experiment treatment
- Both experiment variants maintain consistent telemetry reporting (failedMaybeLater)
bpasero
approved these changes
Feb 18, 2026
bpasero
approved these changes
Feb 18, 2026
Member
|
Thanks! |
chrmarti
approved these changes
Feb 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We see confusion in what Skip for Now means in the Copilot sign-in dialog: https://github.com/github/copilot-growth/issues/773
This PR adds experiment
chatSetupDialogCloseButton:true): show standard X close button, hide Skip for now buttonBoth variants produce the same
failedMaybeLatertelemetry on dismiss.To test locally: set
experiments.override.chatSetupDialogCloseButton: truein settings.Changes
chatSetupRunner.ts: injectIWorkbenchAssignmentService, querychatSetupDialogCloseButtontreatment, conditionally configure dialog close behavior and button set