MQTT Broker Storage Volumes in Kubernetes

Configure initial storage sizes for new deployments and resize existing broker volumes in Kubernetes environments.

The disk space required for MQTT brokers depends on your use case, particularly regarding the use of QoS levels greater than 0, retained messages, and message sizes. Since storage requirements cannot be perfectly predicted, you may need to resize existing volumes or configure appropriate initial sizes.

This document covers both configuring initial storage volume sizes and resizing existing volumes to increase available disk space with minimal service interruption.

The resizing procedure involves removing the StatefulSet, which leaves the broker cluster vulnerable to failures caused by cluster events or human error. Execute this procedure with great care and only in a stable cluster environment.

Configuring Initial Storage Volume Sizes

When deploying Connectware, you can configure the storage volume sizes for the MQTT broker through Helm values. The broker uses two volumes:

Volume	Purpose	Helm Value	Default Size
data	Stores retained messages, offline queues, and cluster metadata	global.broker.storage.data.size	1Gi
log	Stores log files	global.broker.storage.log.size	100Mi

Prerequisites

• Helm version 3 is installed on your system

• The Kubernetes command-line tool kubectl is configured and has access to the target installation

• You know the name and namespace of your Connectware installation. See Obtaining the name, namespace, and version of your Connectware installation

• The values.yaml file is available

Procedure

1. Configure the storage volume sizes by adding the appropriate Helm values to your values.yaml file:

Example

global:
  broker:
    storage:
      data:
        size: 5Gi
      log:
        size: 500Mi

1. Apply the configuration changes using the helm upgrade command:

helm upgrade -n <namespace> <installation-name> -f values.yaml

You cannot change the size of existing volumes through Helm configuration alone. To resize existing volumes, follow the procedure in the next section.

Resizing Existing Storage Volumes

Use this procedure to increase the available disk space for existing broker volumes (PersistentVolumeClaims). This process requires pod restarts, which means clients will need to reconnect.

Prerequisites

• kubectl access to the installation with the current context namespace set to the target namespace (kubectl config set-context --current --namespace <target-namespace>).

• A StorageClass that supports volume expansion. Verify this by running kubectl get sc and checking that ALLOWVOLUMEEXPANSION shows true for the StorageClass used by the volumes.

Preparing the Broker Cluster

1. Ensure you have a healthy broker cluster of at least two pods. Run kubectl get sts broker and verify it shows READY 2/2 or higher with matching numbers on both sides of the slash.

2. If you only have a single broker, scale the StatefulSet to two replicas:

kubectl scale sts broker --replicas 2

1. Export the StatefulSet definition to a local file:

kubectl get sts broker -o yaml > broker.yaml

Resizing Volumes

Repeat this procedure for each broker pod in your cluster.

1. Delete the broker StatefulSet while leaving the pods as orphans:

kubectl delete sts broker --cascade=orphan

1. Set the $broker variable to the pod name of the broker you want to resize (e.g., broker-0):

broker=broker-0

1. Delete the broker pod:

kubectl delete pod $broker

1. Increase the PersistentVolumeClaims size. Replace <size> with the desired Kubernetes quantity for the volume (e.g., 5Gi):

This example resizes the data volume:

kubectl patch pvc brokerdata-$broker --patch '{"spec": { "resources": {"requests": {"storage": "<size>"}}}}'

If you also need to resize the log volume, repeat this step for the log PersistentVolumeClaims:

kubectl patch pvc brokerlog-$broker --patch '{"spec": { "resources": {"requests": {"storage": "<size>"}}}}'

1. Wait until the PersistentVolumeClaims shows the correct capacity:

kubectl get pvc brokerdata-$broker

1. Recreate the StatefulSet:

kubectl apply -f broker.yaml

1. Wait for the StatefulSet to recreate the missing pod. Monitor the status by running kubectl get sts broker until it shows READY 2/2 or higher with matching numbers on both sides of the slash.

2. Verify that all cluster members show consistent cluster information:

kubectl get pod -lapp=broker -o name | xargs -I % kubectl exec % -- vmq-admin cluster show

The output should look similar to this:

Defaulted container "broker" out of: broker, wait-for-k8s (init)
+-------------------------------------------------+---------+
| Node                                            | Running |
+-------------------------------------------------+---------+
| [email protected] | true    |
+-------------------------------------------------+---------+
| [email protected] | true    |
+-------------------------------------------------+---------+
Defaulted container "broker" out of: broker, wait-for-k8s (init)
+-------------------------------------------------+---------+
| Node                                            | Running |
+-------------------------------------------------+---------+
| [email protected] | true    |
+-------------------------------------------------+---------+
| [email protected] | true    |
+-------------------------------------------------+---------+

1. Repeat this procedure for each additional broker pod until all volumes are resized.

Persisting Volume Changes for Future Deployments

After resizing volumes, update your Helm values to reflect the new sizes for future deployments and upgrades. This prevents future Helm upgrades from reverting the volume size.

1. Update the following fields in your values.yaml file based on the volumes you resized:

PersistentVolumeClaims Name	Helm Value
brokerdata-broker-*	global.broker.storage.data.size
brokerlog-broker-*	global.broker.storage.log.size

1. Apply the configuration changes via the helm upgrade command:

helm upgrade -n <namespace> <installation-name> -f values.yaml

For more information, see Applying Helm Configuration Changes.