Skip to content

Separate mock golden files from real GCP files in script_test.go #13478

Description

@maqiuyujoyce

Problem Statement

In tests/e2e/script_test.go (which runs multi-step scenario tests under tests/e2e/testdata/scenarios/), golden file paths are currently hardcoded with identical names for both real GCP and Mock GCP runs:

  • HTTP Logs: _http%02d.log
  • KRM Objects: _object%02d.yaml
  • Exported Objects: _export%d.yaml

When scenario tests are run against Mock GCP (E2E_GCP_TARGET=mock) with WRITE_GOLDEN_OUTPUT=1, mock outputs overwrite the real GCP ground-truth golden files.

Impact

  1. Loss of Ground Truth: Mock reconciliation overwrites authentic GCP logs and object states with simulated mock data.
  2. Inconsistency with Basic Fixtures: In tests/e2e/unified_test.go, basic test fixtures cleanly separate real outputs (_http.log, _generated_object_*.golden.yaml) from mock outputs (_http_mock.log, _http_mock.diff). script_test.go should adhere to the same pattern.
  3. Blocks Automated Parity Verification: Automated alignment testing (pkg/test/goldenalignment) cannot compare multi-step scenario logs and KRM object states side-by-side unless both real and mock files coexist.

Proposed Solution

  1. Target-Aware Filenames in tests/e2e/script_test.go:
    Determine golden file paths dynamically based on E2E_GCP_TARGET:

    • Real GCP (E2E_GCP_TARGET=real):
      • _http%02d.log
      • _object%02d.yaml
      • _export%d.yaml
    • Mock GCP (E2E_GCP_TARGET=mock):
      • _http%02d_mock.log
      • _object%02d_mock.yaml
      • _export%d_mock.yaml
  2. Writing & Comparison Logic:

    • When WRITE_GOLDEN_OUTPUT=1 is set during a mock run, write the step outputs to the *_mock.* files.
    • When running mock assertions, compare against the appropriate golden target files.

Acceptance Criteria

  • Running scenario tests with E2E_GCP_TARGET=real WRITE_GOLDEN_OUTPUT=1 records _http%02d.log and _object%02d.yaml.
  • Running scenario tests with E2E_GCP_TARGET=mock WRITE_GOLDEN_OUTPUT=1 records _http%02d_mock.log and _object%02d_mock.yaml.
  • Mock test runs do not overwrite real GCP golden baseline files in tests/e2e/testdata/scenarios/.
  • Generate and check in *_mock.* golden files for all scenario tests covered by presubmit suites (dev/ci/presubmits/tests-scenarios-suite).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions