Tracks: emit studio_telemetry when analytics opt-in/opt-out changes

[wojtekn]

Related issues

• Fixes STU-2169
• Related to STU-2036

How AI was used in this PR

Claude Code implemented the change end-to-end from the Linear issue. It traced all four analytics-toggle call sites across both renderers, designed the source-attribution plumbing, wrote the unit tests, and fixed the on-transition self-suppression bug found by me during manual testing. I reviewed the code, ran the app to verify the emitted events, and confirmed the ordering logic.

Proposed Changes

The analytics opt-out toggle shipped across onboarding, legacy Settings, and agentic Settings, but flipping it emitted no telemetry — so we had opt-out with no signal on how often users actually opt in or out. This adds a studio_telemetry Tracks event that fires whenever the toggle changes.

The event records:

• status — on / off
• surface — where it was flipped (onboarding / settings)
• ui_version — which renderer (v1 legacy / v2 agentic)

recordTracksEvent is gated by the current opt-out state, so a naive "record then write" would silently drop the very off-transition we care about (the gate the write just closed suppresses it), and a naive "write then record" would drop the on-transition. Recording is ordered around the config write per direction so the event is always emitted while analytics is still ON, and neither transition self-suppresses.

The source attribution is built as a reusable PreferenceChangeSource channel on the preferences write path (not an analytics-only hack), so tracking future settings changes is a small addition in each Main handler rather than new API.

No user-visible behavior change — this is telemetry only, and respects the existing opt-out (opted-out users still emit nothing).

Testing Instructions

1. npm start, then open Settings (agentic or legacy) or onboarding.
2. Toggle Usage statistics off, then on again.
3. In the terminal, confirm two Would have recorded Tracks event: studio_telemetry lines appear — one with status: 'off', one with status: 'on' — each carrying the correct surface and ui_version for where you flipped it.
4. npm test -- apps/studio/src/modules/user-settings/lib/tests/analytics-settings.test.ts — covers both transitions and the record/write ordering.

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

Remaining before merge

• Register studio_telemetry and its eventprops (status, surface, plus common props channel, ui_version, is_a11n, platform, arch, app_version) via the Tracks Registration tool — required for the data to be queryable.

[wpmobilebot]

📊 Performance Test Results

Comparing a3b11c5 vs trunk

app-size

Metric	trunk	a3b11c5	Diff	Change
App Size (Mac)	1378.88 MB	1378.88 MB	+0.00 MB	⚪ 0.0%

site-editor

Metric	trunk	a3b11c5	Diff	Change
load	751 ms	1103 ms	+352 ms	🔴 46.9%

site-startup

Metric	trunk	a3b11c5	Diff	Change
siteCreation	6519 ms	6509 ms	10 ms	⚪ 0.0%
siteStartup	2387 ms	2398 ms	+11 ms	⚪ 0.0%

---

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

[sejas]

The toggle works as expected form the settings modal in Classic and Agentic UI.

@wojtekn , I cannot verify any track event (log) when completing the onboarding. Is that expected?
When completing the onboarding, analyticsOptOut is not saved in shared.json, and therefore there is no opt-out event.

[sejas]

Out of curiosity, why we execute it in different order?

What do you think about this refactor?

Suggested change

	await updateSharedConfig( { analyticsOptOut: false } );
	await recordEvent();
	} else {
	await recordEvent();
	await updateSharedConfig( { analyticsOptOut: true } );
	}
	await updateSharedConfig( { analyticsOptOut: false } );
	} else {
	await updateSharedConfig( { analyticsOptOut: true } );
	}
	await recordEvent();

[wojtekn]

We execute in a different order to track the event when the user switches from disabled to enabled, and vice versa.

[sejas]

I don’t think the order will change what is tracked.

[wojtekn]

@wojtekn , I cannot verify any track event (log) when completing the onboarding. Is that expected?
When completing the onboarding, analyticsOptOut is not saved in shared.json, and therefore there is no opt-out event.

I just tried that having a clean state of the app and it worked fine:

Sentry Logger [log]: [Offline Store]: Popping envelope from offline storage
Would have recorded Tracks event: studio_telemetry {
  platform: 'darwin',
  arch: 'arm64',
  app_version: '1.17.0',
  is_a11n: false,
  channel: 'studio-ui',
  ui_version: 'v1',
  surface: 'onboarding',
  status: 'off'
}
Would have recorded Tracks event: studio_telemetry {
  platform: 'darwin',
  arch: 'arm64',
  app_version: '1.17.0',
  is_a11n: false,
  channel: 'studio-ui',
  ui_version: 'v1',
  surface: 'onboarding',
  status: 'on'
}

Is it possible that you already had it off in config file (set during previous test) and when triggered onboarding, it would change value from off to off, so nothing was logged?

[sejas]

I'll try again and remove previous config.

[sejas]

You're right. It works as expected. I didn't see the checkbox before. I confirm I see the events when clicking the checkbox.

Should we move it above the Log in to WordPress.com.

[wojtekn]

Should we move it above the Log in to WordPress.com.

No, this is the location we already agreed on with the designer.