MQTT Broker Storage Volumes in Kubernetes

Configure initial storage sizes for new deployments and resize existing broker volumes in Kubernetes environments.

The disk space required for MQTT brokers depends on your use case, particularly regarding the use of QoS levels greater than 0, retained messages, and message sizes. Since storage requirements cannot be perfectly predicted, you may need to resize existing volumes or configure appropriate initial sizes.

This document covers both configuring initial storage volume sizes and resizing existing volumes to increase available disk space with minimal service interruption.

The resizing procedure involves removing the StatefulSet, which leaves the broker cluster vulnerable to failures caused by cluster events or human error. Execute this procedure with great care and only in a stable cluster environment.

Configuring Initial Storage Volume Sizes

When deploying Connectware, you can configure the storage volume sizes for the MQTT broker through Helm values. The broker uses two volumes:

Volume	Purpose	Helm Value	Default Size
data	Stores retained messages, offline queues, and cluster metadata	global.broker.storage.data.size	1Gi
log	Stores log files	global.broker.storage.log.size	100Mi

Prerequisites

• Helm version 3 is installed on your system

• The Kubernetes command-line tool kubectl is configured and has access to the target installation

• You know the name and namespace of your Connectware installation. See Obtaining the name, namespace, and version of your Connectware installation

• The values.yaml file is available

Procedure

1. Configure the storage volume sizes by adding the appropriate Helm values to your values.yaml file:

Example

global:
  broker:
    storage:
      data:
        size: 5Gi
      log:
        size: 500Mi

1. Apply the configuration changes using the helm upgrade command:

helm upgrade -n <namespace> <installation-name> -f values.yaml

You cannot change the size of existing volumes through Helm configuration alone. To resize existing volumes, follow the procedure in the next section.

Resizing Existing Broker Storage Volumes

Use this procedure to increase the available disk space for existing broker volumes (PersistentVolumeClaims). This process requires pod restarts, which means clients will need to reconnect.

Prerequisites

• kubectl access to the installation with the current context namespace set to the target namespace (kubectl config set-context --current --namespace <target-namespace>).

• A StorageClass that supports volume expansion. Verify this by running kubectl get sc and checking that ALLOWVOLUMEEXPANSION shows true for the StorageClass used by the volumes.

Preparing the Broker Cluster

1. Ensure you have a healthy broker cluster of at least two pods. Run kubectl get sts broker and verify it shows READY 2/2 or higher with matching numbers on both sides of the slash.

2. If you only have a single broker, scale the StatefulSet to two replicas:

kubectl scale sts broker --replicas 2

1. Export the StatefulSet definition to a local file:

kubectl get sts broker -o yaml > broker.yaml

Resizing Volumes

Repeat this procedure for each broker pod in your cluster.

1. Delete the broker StatefulSet while leaving the pods as orphans:

kubectl delete sts broker --cascade=orphan

1. Set the $broker variable to the pod name of the broker you want to resize (e.g., broker-0):

broker=broker-0

1. Delete the broker pod:

kubectl delete pod $broker

1. Increase the PersistentVolumeClaims size. Replace <size> with the desired Kubernetes quantity for the volume (e.g., 5Gi):

This example resizes the data volume:

kubectl patch pvc brokerdata-$broker --patch '{"spec": { "resources": {"requests": {"storage": "<size>"}}}}'

If you also need to resize the log volume, repeat this step for the log PersistentVolumeClaims:

kubectl patch pvc brokerlog-$broker --patch '{"spec": { "resources": {"requests": {"storage": "<size>"}}}}'

1. Wait until the PersistentVolumeClaims shows the correct capacity:

kubectl get pvc brokerdata-$broker

1. Recreate the StatefulSet:

kubectl apply -f broker.yaml

1. Wait for the StatefulSet to recreate the missing pod. Monitor the status by running kubectl get sts broker until it shows READY 2/2 or higher with matching numbers on both sides of the slash.

2. Verify that all cluster members show consistent cluster information:

kubectl get pod -lapp=broker -o name | xargs -I % kubectl exec % -- vmq-admin cluster show

The output should look similar to this:

Defaulted container "broker" out of: broker, wait-for-k8s (init)
+-------------------------------------------------+---------+
| Node                                            | Running |
+-------------------------------------------------+---------+
| [email protected] | true    |
+-------------------------------------------------+---------+
| [email protected] | true    |
+-------------------------------------------------+---------+
Defaulted container "broker" out of: broker, wait-for-k8s (init)
+-------------------------------------------------+---------+
| Node                                            | Running |
+-------------------------------------------------+---------+
| [email protected] | true    |
+-------------------------------------------------+---------+
| [email protected] | true    |
+-------------------------------------------------+---------+

1. Repeat this procedure for each additional broker pod until all volumes are resized.

Persisting Volume Changes for Future Deployments

After resizing volumes, update your Helm values to reflect the new sizes for future deployments and upgrades. This prevents future Helm upgrades from reverting the volume size.

1. Update the following fields in your values.yaml file based on the volumes you resized:

PersistentVolumeClaims Name	Helm Value
brokerdata-broker-*	global.broker.storage.data.size
brokerlog-broker-*	global.broker.storage.log.size

1. Apply the configuration changes via the helm upgrade command:

helm upgrade -n <namespace> <installation-name> -f values.yaml

For more information, see Applying Helm Configuration Changes.

Deleting Broker Data Volumes

Deleting broker data volumes will remove all retained messages, offline client queues, and cluster metadata. This is typically done when decommissioning an installation or performing a clean reinstall.

This procedure permanently removes all broker data including retained messages, offline queues, and cluster state. This action cannot be undone. Ensure you have proper backups if you need to preserve any data.

Prerequisites

• kubectl access to the installation with the current context namespace set to the target namespace.

• Confirmation that all data can be permanently deleted.

• The broker StatefulSet should be scaled down or deleted before removing volumes.

Procedure

1. Stop the broker cluster by scaling the StatefulSet to zero replicas:

kubectl scale sts broker --replicas 0

1. Wait for all broker pods to terminate:

kubectl get pods -lapp=broker -w

1. List all broker-related PersistentVolumeClaims:

kubectl get pvc | grep broker

1. Delete the broker data volumes. This will remove all data volumes for all broker replicas:

kubectl delete pvc -l app=broker

Alternatively, you can delete specific volumes by name:

kubectl delete pvc brokerdata-broker-0 brokerdata-broker-1
kubectl delete pvc brokerlog-broker-0 brokerlog-broker-1

1. Verify that the PersistentVolumeClaims have been deleted:

kubectl get pvc | grep broker

Redeploying the Broker Cluster

If you plan to redeploy the broker cluster after deleting the data volumes, you can scale it back up with fresh storage:

kubectl scale sts broker --replicas 2

The broker cluster will start with fresh, empty volumes and will need to be reconfigured with any required settings.

Cleaning Up Persistent Volumes

Depending on your storage configuration, the underlying PersistentVolumes might still exist after deleting the claims. To completely remove all storage:

1. List any remaining PersistentVolumes that were bound to the deleted claims:

kubectl get pv | grep broker

1. If PersistentVolumes still exist with a Retain reclaim policy, delete them manually:

kubectl delete pv <persistent-volume-name>

PersistentVolumes with a Delete reclaim policy will be automatically removed when their corresponding PersistentVolumeClaim is deleted.