Progressive Service Deployment

Stagger agent startups to avoid thundering-herd spikes after restarts.

Large deployments with a lot of agents tend to stampede after a restart. Each one immediately tries to fetch its work, causing short, severe load spikes. Progressive service deployment smooths these peaks by introducing short, per-agent delays during startup.

When to Enable Progressive Service Deployment

We recommend turning this on once your deployment reaches ~100,000 resources. At that scale, the stability benefits generally outweigh the slightly slower startup. Below that, it is optional.

Trade-off: Smoother load profile vs. a brief increase in time-to-steady-state after restarts.

When enabled, each agent applies a small, randomized delay before fetch attempts during a warmup window:

• The warmup window lasts for warmupDuration (minutes).

• For each fetch during warmup, the agent waits between warmupMinDelay and warmupMaxDelay (milliseconds).

• After the warmup ends, the delay falls back to the base processing delay (default: 0 ms).

Enabling Progressive Service Deployment

In most cases the default values are sufficient and enough to enable progressive service deployment.

Kubernetes Configuration

Enabling Progressive Service Deployment (Kubernetes)

1. To enable progressive service deployment, set protocolMapperAgentDefaults.progressiveServiceDeployment.enabled to true.

2. Optional: Configure warmup duration and the min/max delay via the corresponding Helm values.

Example

protocolMapperAgentDefaults:
    progressiveServiceDeployment:
        enabled: true
        warmupDuration: 5 # 5 minutes
        warmupMinDelay: 2000 # 2000 milliseconds
        warmupMaxDelay: 10000 # 10000 milliseconds

Docker Compose Configuration

Enabling Progressive Service Deployment (Docker Compose)

1. In the docker-compose.yml file of your agent, set CYBUS_ENABLE_WARMUP_PROGRESSIVE_SERVICE_DEPLOYMENT to true.

2. Optional: Configure warmup duration and the min/max delay via the corresponding environment variables.

Example

services:
    protocol-mapper-agent:
        image: registry.cybus.io/cybus/protocol-mapper:${tag}
        environment:
            CYBUS_AGENT_MODE: distributed
            CYBUS_AGENT_NAME: my-docker-compose-agent
            CYBUS_HOSTNAME_INGRESS: localhost
            CYBUS_ENABLE_WARMUP_PROGRESSIVE_SERVICE_DEPLOYMENT: true
            CYBUS_WARMUP_DURATION_MINUTES: 5
            CYBUS_WARMUP_MINIMUM_DELAY_MS: 2000
            CYBUS_WARMUP_MAXIMUM_DELAY_MS: 10000
        volumes:
            - protocol-mapper-agent:/data
            - ./cybus_combined_ca.crt:/connectware/certs/ca/ca-chain.pem
        restart: unless-stopped
        hostname: my-docker-compose-agent
        network_mode: host
volumes:
    protocol-mapper-agent:

Configuration Parameters

Configuration parameters for the progressive service deployment.

Helm Values

Helm Values (Progressive Service Deployment)

The following Helm values are available in both, protocolMapperAgentDefaults and protocolMapperAgents[*]:

Helm Value	Default	Description
progressiveServiceDeployment.enabled	false	Enables or disables the progressive service deployment warmup mechanism. When enabled, service deployment is gradually ramped up to avoid sudden load. Choices: true, false
progressiveServiceDeployment.warmupDuration	3	Defines the duration (in minutes) for which the warmup mechanism is active. After this time expires, the delay falls back to its default ("0").
progressiveServiceDeployment.warmupMinDelay	500	The minimum (in milliseconds) for the delay an agent can apply during the warmup period.
progressiveServiceDeployment.warmupMaxDelay	800	The maximum (in milliseconds) for the delay an agent can apply during the warmup period.

Environment Variables

Environment Variables Values (Progressive Service Deployment)

Environment Variable	Default	Description
CYBUS_ENABLE_WARMUP_PROGRESSIVE_SERVICE_DEPLOYMENT	false	Enables or disables the progressive service deployment warmup mechanism. When enabled, service deployment is gradually ramped up to avoid sudden load. Choices: true, false
CYBUS_WARMUP_DURATION_MINUTES	3	Defines the duration (in minutes) for which the warmup mechanism is active. After this time expires, the delay falls back to CYBUS_NATS_STREAMS_SERVICESCRUD_PROCESSING_DELAY_MILLISECONDS (default is 0).
CYBUS_WARMUP_MINIMUM_DELAY_MS	500	The minimum (in milliseconds) for the delay an agent can apply during the warmup period.
CYBUS_WARMUP_MAXIMUM_DELAY_MS	800	The maximum (in milliseconds) for the delay an agent can apply during the warmup period.
CYBUS_NATS_STREAMS_SERVICESCRUD_PROCESSING_DELAY_MILLISECONDS	0	The base processing delay (in milliseconds) for NATS Streams Services CRUD operations. This value is used once the warmup duration has expired.