# Progressive Service Deployment

Large deployments with a lot of agents tend to stampede after a restart. Each one immediately tries to fetch its work, causing short, severe load spikes. **Progressive service deployment** smooths these peaks by introducing short, per-agent delays during startup.

## When to Enable Progressive Service Deployment

We recommend turning this on once your deployment reaches **\~100,000 resources**. At that scale, the stability benefits generally outweigh the slightly slower startup. Below that, it is optional.

**Trade-off:** Smoother load profile vs. a brief increase in time-to-steady-state after restarts.

When enabled, each agent applies a small, randomized delay before fetch attempts during a warmup window:

* The warmup window lasts for `warmupDuration` (minutes).
* For each fetch during warmup, the agent waits between `warmupMinDelay` and `warmupMaxDelay` (milliseconds).
* After the warmup ends, the delay falls back to the base processing delay (default: 0 ms).

## Enabling Progressive Service Deployment

In most cases the default values are sufficient and enough to enable progressive service deployment.

{% tabs %}
{% tab title="Kubernetes Configuration" %}

### Enabling Progressive Service Deployment (Kubernetes)

1. To enable progressive service deployment, set `protocolMapperAgentDefaults.progressiveServiceDeployment.enabled` to `true`.
2. Optional: Configure warmup duration and the min/max delay via the [corresponding Helm values](#configuration-parameters).

**Example**

{% code lineNumbers="true" %}

```yaml
protocolMapperAgentDefaults:
  progressiveServiceDeployment:
    enabled: true
    warmupDuration: 5 # 5 minutes
    warmupMinDelay: 2000 # 2000 milliseconds
    warmupMaxDelay: 10000 # 10000 milliseconds
```

{% endcode %}
{% endtab %}

{% tab title="Docker Compose Configuration" %}

### Enabling Progressive Service Deployment (Docker Compose)

1. In the `docker-compose.yml` file of your agent, set `CYBUS_ENABLE_WARMUP_PROGRESSIVE_SERVICE_DEPLOYMENT` to `true`.
2. Optional: Configure warmup duration and the min/max delay via the [corresponding environment variables](#configuration-parameters).

**Example**

{% code lineNumbers="true" %}

```yaml
services:
  protocol-mapper-agent:
    image: registry.cybus.io/cybus/protocol-mapper:${tag}
    environment:
      CYBUS_AGENT_MODE: distributed
      CYBUS_AGENT_NAME: my-docker-compose-agent
      CYBUS_HOSTNAME_INGRESS: localhost
      CYBUS_ENABLE_WARMUP_PROGRESSIVE_SERVICE_DEPLOYMENT: true
      CYBUS_WARMUP_DURATION_MINUTES: 5
      CYBUS_WARMUP_MINIMUM_DELAY_MS: 2000
      CYBUS_WARMUP_MAXIMUM_DELAY_MS: 10000
    volumes:
      - protocol-mapper-agent:/data
      - ./cybus_combined_ca.crt:/connectware/certs/ca/ca-chain.pem
    restart: unless-stopped
    hostname: my-docker-compose-agent
    network_mode: host
volumes:
  protocol-mapper-agent:
```

{% endcode %}
{% endtab %}
{% endtabs %}

## Configuration Parameters

Configuration parameters for the progressive service deployment.

{% tabs %}
{% tab title="Helm Values" %}

### Helm Values (Progressive Service Deployment)

The following Helm values are available in both, `protocolMapperAgentDefaults` and `protocolMapperAgents[*]`:

| Helm Value                                    | Default | Description                                                                                                                                                                               |
| --------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `progressiveServiceDeployment.enabled`        | false   | <p>Enables or disables the progressive service deployment warmup mechanism. When enabled, service deployment is gradually ramped up to avoid sudden load.<br><br>Choices: true, false</p> |
| `progressiveServiceDeployment.warmupDuration` | 3       | Defines the duration (in minutes) for which the warmup mechanism is active. After this time expires, the delay falls back to its default ("0").                                           |
| `progressiveServiceDeployment.warmupMinDelay` | 500     | The minimum (in milliseconds) for the delay an agent can apply during the warmup period.                                                                                                  |
| `progressiveServiceDeployment.warmupMaxDelay` | 800     | The maximum (in milliseconds) for the delay an agent can apply during the warmup period.                                                                                                  |
| {% endtab %}                                  |         |                                                                                                                                                                                           |

{% tab title="Environment Variables" %}

### Environment Variables Values (Progressive Service Deployment)

| Environment Variable                                            | Default | Description                                                                                                                                                                                                  |
| --------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `CYBUS_ENABLE_WARMUP_PROGRESSIVE_SERVICE_DEPLOYMENT`            | false   | <p>Enables or disables the progressive service deployment warmup mechanism. When enabled, service deployment is gradually ramped up to avoid sudden load.<br><br>Choices: true, false</p>                    |
| `CYBUS_WARMUP_DURATION_MINUTES`                                 | 3       | Defines the duration (in minutes) for which the warmup mechanism is active. After this time expires, the delay falls back to `CYBUS_NATS_STREAMS_SERVICESCRUD_PROCESSING_DELAY_MILLISECONDS` (default is 0). |
| `CYBUS_WARMUP_MINIMUM_DELAY_MS`                                 | 500     | The minimum (in milliseconds) for the delay an agent can apply during the warmup period.                                                                                                                     |
| `CYBUS_WARMUP_MAXIMUM_DELAY_MS`                                 | 800     | The maximum (in milliseconds) for the delay an agent can apply during the warmup period.                                                                                                                     |
| `CYBUS_NATS_STREAMS_SERVICESCRUD_PROCESSING_DELAY_MILLISECONDS` | 0       | The base processing delay (in milliseconds) for NATS Streams Services CRUD operations. This value is used once the warmup duration has expired.                                                              |
| {% endtab %}                                                    |         |                                                                                                                                                                                                              |
| {% endtabs %}                                                   |         |                                                                                                                                                                                                              |