Skip Sentry initialization in E2E tests

[bcotrim]

Related issues

• Related to https://github.com/Automattic/dotcom-forge/issues/9704

Proposed Changes

Investigation Process

1. Initially thought the issue was related to E2E tests creating Sentry releases
2. Discovered that dev builds in Buildkite were also creating releases
3. Found that webpack's Sentry plugin was uploading source maps independently of the app's Sentry initialization
4. Identified that the build process needed environment control to prevent unwanted Sentry interactions

Solution

Instead of just preventing Sentry initialization during E2E tests, we implemented a more comprehensive solution:

1. Added DEV_BUILD environment variable to control Sentry source map uploads during builds:

   • Set in Mac builds using export DEV_BUILD=true
   • Set in Windows builds using $env:DEV_BUILD="true"
   • Added to E2E test step as well for consistency

2. Modified webpack plugin configuration to respect this flag:

   process.env.SENTRY_AUTH_TOKEN && !process.env.DEV_BUILD ?
     sentryWebpackPlugin({
       authToken: process.env.SENTRY_AUTH_TOKEN,
       org: 'a8c',
       project: 'studio',
     }) : null

This approach:

• Prevents source map uploads during development builds
• Maintains source map uploads for production releases
• Provides consistent behavior across all platforms
• Keeps the commit hash in production release names for proper error tracking

Note:

• After this PR is approved and merged, I'll proceed with cleaning up Sentry releases as requested in the original issue.

Testing Instructions

1. CI doesn't generate new releases on Sentry anymore for each commit.

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

[ivan-ottinger]

The proposed changes look good and logical to me. 👌🏼

As I am not much familiar with the matter, I will leave approval for others though.

CI doesn't generate new releases on Sentry anymore for each commit.

Are there a specific steps we could take to test the changes?

[bcotrim]

Are there a specific steps we could take to test the changes?

I believe we are able to re-trigger the checks on Buildkite.
If we aren't there's always the option to add a new commit just to trigger a new build.

[sejas]

Thanks for creating the PR. I don't know a proper way to test it so from my point of view is ok to merge the PR and see if we don't upload hash-like versions anymore.

I've noticed that we already have the variable IS_DEV_BUILD for Distribute Dev Builds.

When I investigated this issue in the past, I thought it was due to the Sentry.init calls from NodeJS and the Renderer. But my experiments failed, so thanks for taking this issue.

I executed the Mac universal downloaded from Buildkite and I've seen these Sentry logs in the output, which makes me think this PR is not breaking the Sentry initialization.

Sentry Logger [log]: Starting Electron crashReporter
Sentry Logger [log]: Integration installed: SentryMinidump
Sentry Logger [log]: Integration installed: ElectronBreadcrumbs
Sentry Logger [log]: Integration installed: Net
Sentry Logger [log]: Integration installed: MainContext
Sentry Logger [log]: Integration installed: ChildProcess
Sentry Logger [log]: Integration installed: OnUncaughtException
Sentry Logger [log]: Integration installed: PreloadInjection
Sentry Logger [log]: Integration installed: AdditionalContext
Sentry Logger [log]: Integration installed: Screenshots
Sentry Logger [log]: Integration installed: RendererProfiling
Sentry Logger [log]: Integration installed: InboundFilters
Sentry Logger [log]: Integration installed: FunctionToString
Sentry Logger [log]: Integration installed: LinkedErrors
Sentry Logger [log]: Integration installed: RequestData
Sentry Logger [log]: Integration installed: Console
Sentry Logger [log]: Integration installed: Http
Sentry Logger [log]: Integration installed: Undici
Sentry Logger [log]: Integration installed: OnUnhandledRejection
Sentry Logger [log]: Integration installed: ContextLines
Sentry Logger [log]: Integration installed: LocalVariables
Sentry Logger [log]: Integration installed: Modules
Sentry Logger [log]: Integration installed: MainProcessSession
Sentry Logger [log]: Found previous abnormal session
Sentry Logger [log]: [Tracing] Adding sentry-trace header 56...f7 to outgoing request to "https://pixel.wp.com/b.gif": 
Sentry Logger [log]: [Tracing] Adding sentry-trace header 56...f7 to outgoing request to "https://pixel.wp.com/b.gif": 
Sentry Logger [log]: Successfully sent

[sejas]

We disable Sentry.init for E2E tests in the NodeJS initialization, But I see that DEV_BUILD removes the Sentry plugin from webpack.

studio/src/index.ts

Lines 47 to 48 in 7274053

	! process.env.E2E,
	release: `${ app.getVersion() ? app.getVersion() : COMMIT_HASH }-${ getPlatformName() }`,

[bcotrim]

Unfortunately, the E2E flag only prevents Sentry from being initialized, the source maps are still uploaded, and a new hash-like release is created.
Any action on Sentry that isn't related to a known release will create a new one.

[sejas]

I'm not 100% sure we should add DEV_BUILD to Mac and Windows builds. Are we adding it to avoid Sentry to upload the map?

We should double check if adding DEV_BUILD=true to Mac and Windows build won't affect to Sentry initialization for real users.

[bcotrim]

I'm not 100% sure we should add DEV_BUILD to Mac and Windows builds. Are we adding it to avoid Sentry to upload the map?

Yes, according to Sentry's documentation:

Releases can also be auto created by different systems. For instance upon uploading a source map a release is automatically created. Likewise releases are created by some clients when an event for a release comes in.

[mokagio]

It should be possible to DRY this and other usages in this file at the command level by declaring this env var in the env node below.

env:
  IS_DEV_BUILD: true

That is, unless other calls in the same command need the value unset, because of course setting it at the env level will apply to the whole command execution.

See also:

• https://buildkite.com/docs/pipelines/configure/environment-variables#environment-variable-precedence-job-environment
• https://buildkite.com/docs/pipelines/configure/step-types/command-step#:~:text=env,Example%3A%20RAILS_ENV%3A%20%22test%22

[bcotrim]

Thanks for the review!
I tried to add it to the env variable, but it still triggered the Sentry release (which means it didn't get passed onto the npm commands). For this commit
On Sentry we can see:

I still tried to not repeat the IS_DEV_BUILD variable so much, and did export IS_DEV_BUILD=true before running npm in the command for each of the required steps.
Let me know what you think.

[mokagio]

Hey @bcotrim

I tried to add it to the env variable, but it still triggered the Sentry release (which means it didn't get passed onto the npm commands). For this commit

Oh, that's odd. I'd love to investigate (knowing Buildkite works the way I expect it to is important for my day-to-day) but I don't have the time right now.

Besides, your DRY solution via export does the job nicely.

[mokagio]

This is good to go as far as I'm concerned, but it might be best to wait for a member of your team to give the thumbs up?

Or... maybe it's okay to merge and see what happens. This doesn't affect users and we can iterate on it quickly.

[fredrikekelund]

LGTM 👍 I can confirm that the c8ce53a hash isn't showing up in the Sentry releases list.

[fredrikekelund]

Do we still need to check process.env.E2E in the enabled key just below this?

[bcotrim]

We can probably remove it now.
Locally it will fall into the 'development' check, and on CI we have the IS_DEV_BUILD=true check.
Good catch!