# MQTT Broker Storage Volumes in Kubernetes

The disk space required for MQTT brokers depends on your use case, particularly regarding the use of QoS levels greater than 0, retained messages, and message sizes. Since storage requirements cannot be perfectly predicted, you may need to resize existing volumes or configure appropriate initial sizes.

This document covers both configuring initial storage volume sizes and resizing existing volumes to increase available disk space with minimal service interruption.

{% hint style="warning" %}

### Resizing Volumes Involves Risks

The resizing procedure involves removing the StatefulSet, which leaves the broker cluster vulnerable to failures caused by cluster events or human error. Execute this procedure with great care and only in a stable cluster environment.
{% endhint %}

## Configuring Initial Storage Volume Sizes

When deploying Connectware, you can configure the storage volume sizes for the MQTT broker through Helm values. The broker uses two volumes:

| Volume | Purpose                                                        | Helm Value                        | Default Size |
| ------ | -------------------------------------------------------------- | --------------------------------- | ------------ |
| data   | Stores retained messages, offline queues, and cluster metadata | `global.broker.storage.data.size` | 1Gi          |
| log    | Stores log files                                               | `global.broker.storage.log.size`  | 100Mi        |

**Prerequisites**

* [Helm version 3](https://helm.sh/docs/intro/quickstart/#install-helm) is installed on your system.
* [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) is installed on your system.
* You know the name and namespace of your Connectware installation. See [Obtaining the name, namespace, and version of your Connectware installation](/documentation/connectware-on-kubernetes/connectware-helm-chart.md#obtaining-the-name-namespace-and-version-of-your-connectware-installation).
* The `values.yaml` file is available.

**Procedure**

1. Configure the storage volume sizes by adding the appropriate Helm values to your `values.yaml` file:

**Example**

{% code lineNumbers="true" %}

```yaml
global:
  broker:
    storage:
      data:
        size: 5Gi
      log:
        size: 500Mi
```

{% endcode %}

2. Apply the configuration changes using the `helm upgrade` command:

{% code lineNumbers="true" %}

```bash
helm upgrade -n ${NAMESPACE} ${INSTALLATION_NAME} -f values.yaml
```

{% endcode %}

{% hint style="warning" %}

### Volume Size Changes Require Pod Restarts

You cannot change the size of existing volumes through Helm configuration alone. To resize existing volumes, follow the procedure in the next section.
{% endhint %}

## Resizing Existing Broker Storage Volumes

Use this procedure to increase the available disk space for existing broker volumes (PersistentVolumeClaims). This process requires pod restarts, which means clients will need to reconnect.

### Prerequisites

* kubectl has access to the installation with the current context namespace set to the target namespace (`kubectl config set-context --current --namespace ${NAMESPACE}`).
* A `StorageClass` that supports volume expansion. Verify this by running `kubectl get sc` and checking that `ALLOWVOLUMEEXPANSION` shows `true` for the StorageClass used by the volumes.

### Preparing the Broker Cluster

1. Ensure you have a healthy broker cluster of at least two pods. Run `kubectl get sts broker` and verify it shows `READY 2/2` or higher with matching numbers on both sides of the slash.
2. If you only have a single broker, scale the StatefulSet to two replicas:

{% code lineNumbers="true" %}

```bash
kubectl scale sts broker --replicas 2
```

{% endcode %}

3. Export the StatefulSet definition to a local file:

{% code lineNumbers="true" %}

```bash
kubectl get sts broker -o yaml > broker.yaml
```

{% endcode %}

### Resizing Volumes

Repeat this procedure for each broker pod in your cluster.

1. Delete the broker StatefulSet while leaving the pods as orphans:

{% code lineNumbers="true" %}

```bash
kubectl delete sts broker --cascade=orphan
```

{% endcode %}

2. Set the `$broker` variable to the pod name of the broker you want to resize (e.g., `broker-0`):

{% code lineNumbers="true" %}

```bash
broker=broker-0
```

{% endcode %}

3. Delete the broker pod:

{% code lineNumbers="true" %}

```bash
kubectl delete pod $broker
```

{% endcode %}

4. Increase the PersistentVolumeClaims size. Replace `<size>` with the desired Kubernetes quantity for the volume (e.g., `5Gi`):

This example resizes the data volume:

{% code lineNumbers="true" %}

```bash
kubectl patch pvc brokerdata-$broker --patch '{"spec": { "resources": {"requests": {"storage": "<size>"}}}}'
```

{% endcode %}

If you also need to resize the log volume, repeat this step for the log PersistentVolumeClaims:

{% code lineNumbers="true" %}

```bash
kubectl patch pvc brokerlog-$broker --patch '{"spec": { "resources": {"requests": {"storage": "<size>"}}}}'
```

{% endcode %}

5. Wait until the PersistentVolumeClaims shows the correct capacity:

{% code lineNumbers="true" %}

```bash
kubectl get pvc brokerdata-$broker
```

{% endcode %}

6. Recreate the StatefulSet:

{% code lineNumbers="true" %}

```bash
kubectl apply -f broker.yaml
```

{% endcode %}

7. Wait for the StatefulSet to recreate the missing pod. Monitor the status by running `kubectl get sts broker` until it shows `READY 2/2` or higher with matching numbers on both sides of the slash.
8. Verify that all cluster members show consistent cluster information:

{% code lineNumbers="true" %}

```bash
kubectl get pod -lapp=broker -o name | xargs -I % kubectl exec % -- vmq-admin cluster show
```

{% endcode %}

The output should look similar to this:

{% code lineNumbers="true" %}

```
Defaulted container "broker" out of: broker, wait-for-k8s (init)
+-------------------------------------------------+---------+
| Node                                            | Running |
+-------------------------------------------------+---------+
| VerneMQ@broker-0.broker.cybus.svc.cluster.local | true    |
+-------------------------------------------------+---------+
| VerneMQ@broker-1.broker.cybus.svc.cluster.local | true    |
+-------------------------------------------------+---------+
Defaulted container "broker" out of: broker, wait-for-k8s (init)
+-------------------------------------------------+---------+
| Node                                            | Running |
+-------------------------------------------------+---------+
| VerneMQ@broker-0.broker.cybus.svc.cluster.local | true    |
+-------------------------------------------------+---------+
| VerneMQ@broker-1.broker.cybus.svc.cluster.local | true    |
+-------------------------------------------------+---------+
```

{% endcode %}

9. Repeat this procedure for each additional broker pod until all volumes are resized.

### Persisting Volume Changes for Future Deployments

After resizing volumes, update your Helm values to reflect the new sizes for future deployments and upgrades. This prevents future Helm upgrades from reverting the volume size.

1. Update the following fields in your `values.yaml` file based on the volumes you resized:

| PersistentVolumeClaims Name | Helm Value                        |
| --------------------------- | --------------------------------- |
| brokerdata-broker-\*        | `global.broker.storage.data.size` |
| brokerlog-broker-\*         | `global.broker.storage.log.size`  |

2. Apply the configuration changes via the `helm upgrade` command:

{% code lineNumbers="true" %}

```bash
helm upgrade -n ${NAMESPACE} ${INSTALLATION_NAME} -f values.yaml
```

{% endcode %}

For more information, see [Applying Helm Configuration Changes](/documentation/connectware-on-kubernetes/connectware-helm-chart.md#applying-helm-configuration-changes).

## Deleting Broker Data Volumes

To permanently delete broker data volumes, see [Deleting Broker Data Volumes](/broker/cybusmq/operations/deleting-broker-data-volumes.md).


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.cybus.io/broker/cybusmq/operations/mqtt-broker-storage-volumes-in-kubernetes.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.