Problem Statement
In tests/e2e/script_test.go (which runs multi-step scenario tests under tests/e2e/testdata/scenarios/), golden file paths are currently hardcoded with identical names for both real GCP and Mock GCP runs:
- HTTP Logs:
_http%02d.log
- KRM Objects:
_object%02d.yaml
- Exported Objects:
_export%d.yaml
When scenario tests are run against Mock GCP (E2E_GCP_TARGET=mock) with WRITE_GOLDEN_OUTPUT=1, mock outputs overwrite the real GCP ground-truth golden files.
Impact
- Loss of Ground Truth: Mock reconciliation overwrites authentic GCP logs and object states with simulated mock data.
- Inconsistency with Basic Fixtures: In
tests/e2e/unified_test.go, basic test fixtures cleanly separate real outputs (_http.log, _generated_object_*.golden.yaml) from mock outputs (_http_mock.log, _http_mock.diff). script_test.go should adhere to the same pattern.
- Blocks Automated Parity Verification: Automated alignment testing (
pkg/test/goldenalignment) cannot compare multi-step scenario logs and KRM object states side-by-side unless both real and mock files coexist.
Proposed Solution
-
Target-Aware Filenames in tests/e2e/script_test.go:
Determine golden file paths dynamically based on E2E_GCP_TARGET:
- Real GCP (
E2E_GCP_TARGET=real):
_http%02d.log
_object%02d.yaml
_export%d.yaml
- Mock GCP (
E2E_GCP_TARGET=mock):
_http%02d_mock.log
_object%02d_mock.yaml
_export%d_mock.yaml
-
Writing & Comparison Logic:
- When
WRITE_GOLDEN_OUTPUT=1 is set during a mock run, write the step outputs to the *_mock.* files.
- When running mock assertions, compare against the appropriate golden target files.
Acceptance Criteria
Problem Statement
In
tests/e2e/script_test.go(which runs multi-step scenario tests undertests/e2e/testdata/scenarios/), golden file paths are currently hardcoded with identical names for both real GCP and Mock GCP runs:_http%02d.log_object%02d.yaml_export%d.yamlWhen scenario tests are run against Mock GCP (
E2E_GCP_TARGET=mock) withWRITE_GOLDEN_OUTPUT=1, mock outputs overwrite the real GCP ground-truth golden files.Impact
tests/e2e/unified_test.go, basic test fixtures cleanly separate real outputs (_http.log,_generated_object_*.golden.yaml) from mock outputs (_http_mock.log,_http_mock.diff).script_test.goshould adhere to the same pattern.pkg/test/goldenalignment) cannot compare multi-step scenario logs and KRM object states side-by-side unless both real and mock files coexist.Proposed Solution
Target-Aware Filenames in
tests/e2e/script_test.go:Determine golden file paths dynamically based on
E2E_GCP_TARGET:E2E_GCP_TARGET=real):_http%02d.log_object%02d.yaml_export%d.yamlE2E_GCP_TARGET=mock):_http%02d_mock.log_object%02d_mock.yaml_export%d_mock.yamlWriting & Comparison Logic:
WRITE_GOLDEN_OUTPUT=1is set during a mock run, write the step outputs to the*_mock.*files.Acceptance Criteria
E2E_GCP_TARGET=real WRITE_GOLDEN_OUTPUT=1records_http%02d.logand_object%02d.yaml.E2E_GCP_TARGET=mock WRITE_GOLDEN_OUTPUT=1records_http%02d_mock.logand_object%02d_mock.yaml.tests/e2e/testdata/scenarios/.*_mock.*golden files for all scenario tests covered by presubmit suites (dev/ci/presubmits/tests-scenarios-suite).