Progressive Service Deployment

Stagger agent startups to avoid thundering-herd spikes after restarts.

Large deployments with a lot of agents tend to stampede after a restart. Each one immediately tries to fetch its work, causing short, severe load spikes. Progressive service deployment smooths these peaks by introducing short, per-agent delays during startup.

When to Enable Progressive Service Deployment

We recommend turning this on once your deployment reaches ~100,000 resources. At that scale, the stability benefits generally outweigh the slightly slower startup. Below that, it is optional.

Trade-off: Smoother load profile vs. a brief increase in time-to-steady-state after restarts.

When enabled, each agent applies a small, randomized delay before fetch attempts during a warmup window:

• The warmup window lasts for warmupDuration (minutes).

• For each fetch during warmup, the agent waits between warmupMinDelay and warmupMaxDelay (milliseconds).

• After the warmup ends, the delay falls back to the base processing delay (default: 0 ms).

Enabling Progressive Service Deployment

In most cases the default values are sufficient and enough to enable progressive service deployment.

Kubernetes Configuration

Enabling Progressive Service Deployment (Kubernetes)

1. To enable progressive service deployment, set protocolMapperAgentDefaults.progressiveServiceDeployment.enabled to true.

2. Optional: Configure warmup duration and the min/max delay via the corresponding Helm values.

Example

protocolMapperAgentDefaults:
  progressiveServiceDeployment:
    enabled: true
    warmupDuration: 5 # 5 minutes
    warmupMinDelay: 2000 # 2000 milliseconds
    warmupMaxDelay: 10000 # 10000 milliseconds

Docker Compose Configuration

Configuration Parameters

Configuration parameters for the progressive service deployment.

Helm Values

Helm Values (Progressive Service Deployment)

The following Helm values are available in both, protocolMapperAgentDefaults and protocolMapperAgents[*]:

Helm Value	Default	Description
progressiveServiceDeployment.enabled	false	Enables or disables the progressive service deployment warmup mechanism. When enabled, service deployment is gradually ramped up to avoid sudden load. Choices: true, false
progressiveServiceDeployment.warmupDuration	3	Defines the duration (in minutes) for which the warmup mechanism is active. After this time expires, the delay falls back to its default ("0").
progressiveServiceDeployment.warmupMinDelay	500	The minimum (in milliseconds) for the delay an agent can apply during the warmup period.
progressiveServiceDeployment.warmupMaxDelay	800	The maximum (in milliseconds) for the delay an agent can apply during the warmup period.

Environment Variables