Skip to content

ComputeRoute: Fix network and next_hop_gateway diff discrepancies between tf and direct controllers #13581

Description

@anhdle-sso

Bug Description

In the 1.157 preview evaluation comparing the tf (default) and direct controllers for compute.cnrm.cloud.google.com/ComputeRoute, all 51 evaluated instances (100%) report no diff_fields under the tf controller, but report diff_fields (network, next_hop_gateway) under the direct controller (currently marked RECONCILE_STATUS_HEALTHY because ComputeRoute does not support in-place updates in GCP, so the diff is ignored/allowed during reconcile, but a field-level diff still exists).

Preview Summary (result_1.157)

Controller Total Healthy Unhealthy % Healthy % Unhealthy With diff_fields (Healthy / Unhealthy)
direct 51 51 0 100.00% 0.00% 51 (51 H / 0 U)
tf 51 51 0 100.00% 0.00% 0 (0 H / 0 U)

Discrepant diff_fields in direct Controller

diff_field Total Occurrences In UNHEALTHY In HEALTHY (Ignored/Allowed) % of Controller Total % of Controller Unhealthy
network 51 0 51 100.00% 0.00%
next_hop_gateway 51 0 51 100.00% 0.00%

Observed Diff Combinations (tf vs. direct)

Count tf Status & diff_fields direct Status & diff_fields
51 Healthy: (no diff_fields) Healthy: network, next_hop_gateway

Workflow & Instructions

Maintain a journal file (e.g., journal_computeroute.md) throughout the entire effort to track all replication attempts, test runs, newly discovered field discrepancies, and fixes.

Step 1: Replication of the Bugs

  1. Run Existing Tests:
    • Run TestMigrationToDirect for the existing tests for ComputeRoute.
  2. Verify Replication in Logs:
    • Check the _migration_diffs.json file of each test. The issue is replicated once _migration_diffs.json has an entry for "controller": "direct", and the fields that have diffs (network, next_hop_gateway) show up in the entry.
    • If the resource's direct controller supports updates in GCP, the Phase 3 log _http_migration_phase3_direct_takeover.log should also contain a non-GET call.
  3. Create New Tests (If Needed):
    • If the existing tests cannot reproduce the issue, create new tests covering network and nextHopGateway and run TestMigrationToDirect with real GCP to reproduce the problem.
  4. Commit or Stop:
    • If the problem is successfully reproduced: Create a commit with the reproduction test(s), test logs, and journal file, then proceed to Step 2.
    • If the problem cannot be reproduced: Commit the test logs and journal file, comment on the PR that the issue cannot be reproduced, stop, and create the PR.

Step 2: Fix the Discrepancies and Verify

  1. Implement the Fix:
    • Once the problem is successfully reproduced, create a fix so the direct controller and tf controller no longer have a logic/normalization discrepancy for network and next_hop_gateway (e.g., full URL vs. partial selfLink / relative resource name normalization).
  2. Handle Additional Field Discrepancies:
    • If you find other fields with discrepancies during testing, create a test case that reproduces each newly discovered issue in a separate commit, and fix them.
  3. Re-run and Verify Clean Migration Logs:
    • Once all discrepancies are discovered and fixed, re-run TestMigrationToDirect and keep the new test logs, which must show:
      • No migration diff in _migration_diffs.json.
      • No non-GET call in _http_migration_phase3_direct_takeover.log during Phase 3.
  4. Create Separate Fix PR:
    • Put the controller fixes, the updated clean test logs, and the journal file in a separate PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions