[TT-14618] Fix SSL certificate loading from MDCB at startup

[edsonmichaque]

Description

Data plane gateways fail to load SSL certificates from MDCB during startup in v5.8.0+. This PR fixes the regression by adding exponential backoff retry logic to certificate loading.

Related Issue

https://tyktech.atlassian.net/browse/TT-14618

Motivation and Context

Critical regression from TT-14163 (PR #6910). Gateway starts in emergency mode immediately, blocking certificate fetch before RPC ready. Affects v5.8.0+.

How This Has Been Tested

Unit test in certs/manager_tt14618_test.go demonstrates 6 retry attempts over 0.8s, successfully loading certificate after RPC becomes ready.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist

• I ensured that the documentation is up to date
• I explained why this PR updates go.mod in detail with reasoning why it's required
• I would like a code coverage CI quality gate exception and have explained why

Ticket Details

TT-14618

Status	Ready for Dev
Summary	[regression]Data plane gateways cannot start if they're configured with a TYK_GW_HTTPSERVEROPTIONS_SSLCERTIFICATES that is in the certificate store

Generated at: 2026-01-31 10:33:13

[probelabs]

This PR addresses a critical regression where data plane gateways fail to start if configured to load SSL certificates from a Multi-Data Center Bridge (MDCB). The issue stems from a race condition during startup: the gateway attempts to fetch certificates before the RPC connection to MDCB is fully established, resulting in an immediate failure.

The fix introduces a resilient, configurable exponential backoff retry mechanism into the certificate loading process. When the CertificateManager fails to fetch a certificate due to a storage.ErrMDCBConnectionLost error, it now automatically retries the operation with increasing delays. This gives the RPC connection sufficient time to initialize, making the gateway startup sequence more robust.

Files Changed Analysis

The changes are well-contained and focused on implementing and configuring the new retry logic:

• certs/manager.go: Contains the core logic change. The List function now uses the cenkalti/backoff/v4 library to wrap the storage.GetKey call in a retry loop, which is triggered specifically by the storage.ErrMDCBConnectionLost error.
• certs/cert_retry_test.go: A comprehensive new test file (+885 lines) has been added to rigorously validate the retry mechanism. It covers various scenarios, including flaky connections, multiple failures, and performance at scale, ensuring the logic is correct and efficient.
• config/config.go & cli/linter/schema.json: New configuration options (e.g., RPCCertFetchMaxElapsedTime, RPCCertFetchMaxRetries) are added to the SlaveOptionsConfig struct and the linter schema, allowing operators to fine-tune the retry behavior.
• gateway/server.go: The gateway's startup logic is updated to read the new configuration values, provide sensible defaults, and pass them to the CertificateManager during its initialization.
• certs/manager_test.go & gateway/server_test.go: Existing tests are updated to accommodate the new function signatures and configuration defaults.

Architecture & Impact Assessment

• What this PR accomplishes: It resolves a critical startup failure for gateways in an MDCB topology, significantly improving the reliability and robustness of the data plane's initialization sequence.
• Key technical changes introduced:
  1. Exponential Backoff: Implements a blocking retry mechanism to handle the specific ErrMDCBConnectionLost error during certificate loading.
  2. Configurability: Exposes the retry strategy parameters through the main configuration, providing safe defaults while allowing operators to tune timings and retry counts.
• Affected system components: The primary impact is on the CertificateManager and the gateway's startup configuration process. This change hardens the interaction between the gateway and the MDCB storage layer for certificate retrieval.

Certificate Loading Flow with Retry Logic

sequenceDiagram
    participant Gateway Startup
    participant CertificateManager
    participant MDCB Storage

    Gateway Startup->>CertificateManager: List(certIDs)
    CertificateManager->>MDCB Storage: GetKey("raw-cert-1")
    MDCB Storage-->>CertificateManager: Fail (ErrMDCBConnectionLost)
    
    Note over CertificateManager: MDCB not ready, initiating exponential backoff.

    loop Retry Loop (until success or timeout)
        CertificateManager->>MDCB Storage: GetKey("raw-cert-1")
        alt Connection still fails
            MDCB Storage-->>CertificateManager: Fail (ErrMDCBConnectionLost)
            Note over CertificateManager: Wait for next interval...
        else Connection succeeds
            MDCB Storage-->>CertificateManager: Success (Certificate Data)
            break
        end
    end

    Note over CertificateManager: MDCB connection is now stable.
    loop For remaining certs
        CertificateManager->>MDCB Storage: GetKey("raw-cert-N")
        MDCB Storage-->>CertificateManager: Success (Certificate Data)
    end
    end
    CertificateManager-->>Gateway Startup: Return all loaded certificates

Loading

Scope Discovery & Context Expansion

The fix is precisely targeted at the CertificateManager, which is the correct component to handle this logic as it is responsible for retrieving certificates from storage. The error being handled, storage.ErrMDCBConnectionLost, is specific to the RPCStorageHandler when the gateway's RPC connection is unavailable, confirming this is a well-scoped solution.

This PR establishes a robust pattern for handling startup race conditions involving MDCB. The retry logic implemented here could serve as a valuable blueprint for hardening other parts of the system that exhibit similar dependencies on the RPC connection being available at startup.

Metadata

• Review Effort: 3 / 5
• Primary Label: bug

---

Powered by Visor from Probelabs

Last updated: 2026-01-31T10:34:39.996Z | Triggered by: pr_updated | Commit: d4a2b0d

💡 TIP: You can chat with Visor using /visor ask <your question>

[probelabs]

✅ Security Check Passed

No security issues found – changes LGTM.

Architecture Issues (2)

Severity	Location	Issue
🟡 Warning	config/config.go:449	The comment for `RPCCertFetchMaxRetries` states that the default is 3, but the actual default value implemented in `certs/manager.go` and applied in `gateway/server.go` is 5. This can be misleading for users configuring the gateway. 💡 Suggestion Update the comment to reflect the correct default value of 5 to ensure documentation is consistent with the implementation. 🔧 Suggested Fix // RPCCertFetchMaxRetries sets the maximum number of retry attempts for certificate fetch. 0 means unlimited (time-based only) (default: 5)
🟡 Warning	certs/manager.go:91-99	The `NewCertificateManager` function sets default values for the backoff timing parameters by hardcoding them (e.g., `30 * time.Second`). However, these defaults are already defined as constants (e.g., `DefaultRPCCertFetchMaxElapsedTime`) in the same file and are used elsewhere for setting defaults. This creates a duplicate source for default values and is inconsistent with how defaults are handled in the gateway. 💡 Suggestion Replace the hardcoded default values in `NewCertificateManager` with the corresponding constants defined in the `certs` package to maintain a single source of truth and improve consistency. 🔧 Suggested Fix if certFetchMaxElapsedTime == 0 { certFetchMaxElapsedTime = DefaultRPCCertFetchMaxElapsedTime } if certFetchInitialInterval == 0 { certFetchInitialInterval = DefaultRPCCertFetchInitialInterval } if certFetchMaxInterval == 0 { certFetchMaxInterval = DefaultRPCCertFetchMaxInterval }

Performance Issues (1)

Severity	Location	Issue
🟢 Info	certs/manager.go:466-549	The function fetches multiple certificates sequentially within a loop. While this is safe and avoids a "thundering herd" problem during initial connection failures, it leads to serialized network requests when the connection is healthy. For a large number of certificates, this sequential processing can be significantly slower than a parallel approach. 💡 Suggestion Consider optimizing this by first performing a single, retry-enabled connection probe (e.g., using the `Exists` method on the storage handler) to ensure the storage is available. Once the connection is established, the certificates can be fetched concurrently using goroutines. This would significantly reduce the total time required for network I/O, especially when loading many certificates over a network with non-trivial latency.

Quality Issues (2)

Severity	Location	Issue
🟡 Warning	certs/manager.go:480-486	The error handling within the `backoff.Retry` operation is implicit. It relies on a variable (`err`) from the parent scope to propagate non-retryable errors. The operation returns `nil` for non-retryable errors, which signals success to the `backoff` library, and the error is then handled outside the retry block. This pattern is functional but can be confusing and fragile during refactoring. 💡 Suggestion To make the intent clearer and the error handling more robust, use `backoff.Permanent(err)` to explicitly mark an error as non-retryable. This causes `backoff.Retry` to stop immediately and return the permanent error, simplifying the control flow.
🟡 Warning	config/config.go:455	The comment for `RPCCertFetchMaxRetries` states that the default value is 3, but the implementation in `gateway/server.go` and the associated tests use a default of 5, as defined by `certs.DefaultRPCCertFetchMaxRetries`. 💡 Suggestion Update the comment to reflect the correct default value of 5 to avoid confusion for operators configuring the gateway. 🔧 Suggested Fix // RPCCertFetchMaxRetries sets the maximum number of retry attempts for certificate fetch. 0 means unlimited (time-based only) (default: 5)

---

Powered by Visor from Probelabs

Last updated: 2026-01-31T10:34:42.778Z | Triggered by: pr_updated | Commit: d4a2b0d

💡 TIP: You can chat with Visor using /visor ask <your question>

[github-actions]

API Changes

--- prev.txt	2026-01-31 10:33:31.118194109 +0000
+++ current.txt	2026-01-31 10:33:21.116991224 +0000
@@ -5614,6 +5614,40 @@
 package certs // import "github.com/TykTechnologies/tyk/certs"
 
 
+CONSTANTS
+
+const (
+
+	// MDCB retry configuration for certificate loading during startup.
+	//
+	// When the gateway starts with MDCB configured, it enters emergency mode until the
+	// RPC connection is established. During this period, certificate fetches from MDCB
+	// will fail with ErrMDCBConnectionLost. These constants control the retry behavior:
+	//
+	// DefaultRPCCertFetchMaxElapsedTime: Maximum time to wait for MDCB to become ready (30 seconds)
+	//   - If MDCB doesn't respond within this window, we give up and fall back to local files
+	//   - This prevents indefinite blocking during gateway startup
+	//
+	// DefaultRPCCertFetchInitialInterval: Starting delay for exponential backoff (100ms)
+	//   - First retry happens after 100ms
+	//   - Each subsequent retry doubles: 100ms → 200ms → 400ms → 800ms → 1600ms → 2000ms (capped)
+	//
+	// DefaultRPCCertFetchMaxInterval: Maximum delay between retry attempts (2 seconds)
+	//   - Caps exponential growth to prevent excessively long waits
+	//   - Ensures we check MDCB at least every 2 seconds
+	DefaultRPCCertFetchMaxElapsedTime  = 30 * time.Second
+	DefaultRPCCertFetchInitialInterval = 100 * time.Millisecond
+	DefaultRPCCertFetchMaxInterval     = 2 * time.Second
+
+	// DefaultRPCCertFetchRetryEnabled: Enable retry by default (true)
+	DefaultRPCCertFetchRetryEnabled = true
+
+	// DefaultRPCCertFetchMaxRetries: Maximum number of retry attempts (5)
+	//   - 0 means unlimited (time-based only)
+	//   - Default of 5 provides reasonable retry attempts with exponential backoff
+	DefaultRPCCertFetchMaxRetries = 5
+)
+
 VARIABLES
 
 var (
@@ -5630,7 +5664,7 @@
 func GetCertIDAndChainPEM(certData []byte, secret string) (string, []byte, error)
 func ParsePEM(data []byte, secret string) ([]*pem.Block, error)
 func ParsePEMCertificate(data []byte, secret string) (*tls.Certificate, error)
-func NewCertificateManager(storage storage.Handler, secret string, logger *logrus.Logger, migrateCertList bool) *certificateManager
+func NewCertificateManager(storageHandler storage.Handler, secret string, logger *logrus.Logger, migrateCertList bool, certFetchMaxElapsedTime, certFetchInitialInterval, certFetchMaxInterval time.Duration, certFetchRetryEnabled bool, certFetchMaxRetries int) *certificateManager
 func NewSlaveCertManager(localStorage, rpcStorage storage.Handler, secret string, logger *logrus.Logger, migrateCertList bool) *certificateManager
 
 TYPES
@@ -7369,6 +7403,21 @@
 	// RPCKeysCacheExpiration defines the expiration time of the rpc cache that stores the keys, defined in seconds
 	RPCGlobalCacheExpiration float32 `json:"rpc_global_cache_expiration"`
 
+	// RPCCertFetchMaxElapsedTime sets the maximum time in seconds to retry certificate fetch from MDCB during startup (default: 30)
+	RPCCertFetchMaxElapsedTime float32 `json:"rpc_cert_fetch_max_elapsed_time"`
+
+	// RPCCertFetchInitialInterval sets the initial retry interval in seconds for certificate fetch backoff (default: 0.1)
+	RPCCertFetchInitialInterval float32 `json:"rpc_cert_fetch_initial_interval"`
+
+	// RPCCertFetchMaxInterval sets the maximum retry interval in seconds for certificate fetch backoff (default: 2)
+	RPCCertFetchMaxInterval float32 `json:"rpc_cert_fetch_max_interval"`
+
+	// RPCCertFetchRetryEnabled enables exponential backoff retry for certificate fetch from MDCB during startup (default: true)
+	RPCCertFetchRetryEnabled *bool `json:"rpc_cert_fetch_retry_enabled"`
+
+	// RPCCertFetchMaxRetries sets the maximum number of retry attempts for certificate fetch. 0 means unlimited (time-based only) (default: 3)
+	RPCCertFetchMaxRetries *int `json:"rpc_cert_fetch_max_retries"`
+
 	// SynchroniserEnabled enable this config if MDCB has enabled the synchoniser. If disabled then it will ignore signals to synchonise recources
 	SynchroniserEnabled bool `json:"synchroniser_enabled"`
 

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
84.0% Coverage on New Code
0.2% Duplication on New Code

See analysis details on SonarQube Cloud

[edsonmichaque]

use consts, not magic values

[edsonmichaque]

use consts, not magic values

[edsonmichaque]

use consts, not magic values

[edsonmichaque]

use same pattern for initialization?