MQTT Broker Storage Volumes in Kubernetes

Configure initial storage sizes for new deployments and resize existing broker volumes in Kubernetes environments.

The broker uses two persistent volumes: one for queue data and cluster metadata, and one for log files. The disk space required depends on your use case, particularly the use of QoS levels greater than zero, retained messages, and message sizes. Since storage requirements cannot be perfectly predicted, you can configure initial volume sizes at install time and resize volumes in a running deployment when needed.

Configuring Initial Storage Volume Sizes

When deploying Connectware, you can configure the storage volume sizes and StorageClass for the Broker through Helm values:

Helm Value	Description	Default
broker.persistence.data.size	Storage for retained messages, offline queues, and cluster metadata	1Gi
broker.persistence.data.storageClassName	StorageClass for the data volume	"" (cluster default)
broker.persistence.log.size	Storage for broker log files	200Mi
broker.persistence.log.storageClassName	StorageClass for the log volume	"" (cluster default)

Configure the storage volume sizes by adding the appropriate Helm values to your values.yaml file:

values.yaml

broker:
  persistence:
    data:
      size: 5Gi
    log:
      size: 500Mi

Because this example does not specify storageClassName, Kubernetes uses the cluster default StorageClass for both volumes.

Resizing Existing Broker Storage Volumes

Use this procedure to increase the available disk space for existing broker volumes (PersistentVolumeClaims). This process requires pod restarts, which means clients must reconnect.

Resizing volumes requires pod restarts and involves risks

Resizing broker volumes requires deleting and recreating broker pods. Clients are disconnected during the process and must reconnect once the pods are back. The procedure also involves removing the StatefulSet, which leaves the broker cluster vulnerable to failures caused by cluster events or human error. Execute this procedure with great care and only in a stable cluster environment.

Prerequisites

• kubectl is installed on your system.

• You know the namespace and name of your Connectware installation, referred to as ${NAMESPACE} and ${INSTALLATION_NAME} throughout this guide.

• The used StorageClass supports volume expansion. Run kubectl get sc and confirm that ALLOWVOLUMEEXPANSION is true for that StorageClass.

Preparing the Broker Cluster

1. Ensure you have a healthy broker cluster of at least two pods. Run kubectl get sts broker -n ${NAMESPACE} and verify it shows READY 2/2 or higher with matching numbers on both sides of the slash.

2. If you only have a single broker, scale the StatefulSet to two replicas:

kubectl scale sts broker --replicas 2 -n ${NAMESPACE}

1. Export the StatefulSet definition to a local file:

kubectl get sts broker -n ${NAMESPACE} -o yaml > broker.yaml

Resizing Volumes

Repeat this procedure for each broker pod in your cluster.

1. Delete the broker StatefulSet while leaving the pods as orphans:

kubectl delete sts broker -n ${NAMESPACE} --cascade=orphan

1. Set the $broker variable to the pod name of the broker you want to resize (for example, broker-0):

broker=broker-0

1. Delete the broker pod:

kubectl delete pod $broker -n ${NAMESPACE}

1. Increase the PersistentVolumeClaims size. Replace ${SIZE} with the desired Kubernetes quantity for the volume (for example, 5Gi):

To resize the data volume:

kubectl patch pvc brokerdata-$broker -n ${NAMESPACE} --patch '{"spec": { "resources": {"requests": {"storage": "${SIZE}"}}}}'

To resize the log volume:

kubectl patch pvc brokerlog-$broker -n ${NAMESPACE} --patch '{"spec": { "resources": {"requests": {"storage": "${SIZE}"}}}}'

1. Wait until the PersistentVolumeClaims show the correct capacity:

kubectl get pvc brokerdata-$broker -n ${NAMESPACE}

1. Recreate the StatefulSet:

kubectl apply -f broker.yaml -n ${NAMESPACE}

1. Wait for the StatefulSet to recreate the missing pod. Monitor the status by running kubectl get sts broker -n ${NAMESPACE} until it shows READY 2/2 or higher with matching numbers on both sides of the slash.

2. Verify that all cluster members show consistent cluster information:

kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/name=broker -o name | xargs -I % kubectl exec -n ${NAMESPACE} % -- vmq-admin cluster show

The output should list all broker nodes as Running: true on each pod where you run the command.

+------------------------------------------------------------------------------+---------+
| Node                                                                         | Running |
+------------------------------------------------------------------------------+---------+
| [e-mail]    | true    |
+------------------------------------------------------------------------------+---------+
| [e-mail]    | true    |
+------------------------------------------------------------------------------+---------+
+------------------------------------------------------------------------------+---------+
| Node                                                                         | Running |
+------------------------------------------------------------------------------+---------+
| [e-mail]    | true    |
+------------------------------------------------------------------------------+---------+
| [e-mail]    | true    |
+------------------------------------------------------------------------------+---------+

1. Repeat this procedure for each additional broker pod until all volumes are resized.

Persisting Volume Changes for Future Deployments

After resizing volumes, update your Helm values to reflect the new sizes for future deployments and upgrades. This prevents future Helm upgrades from reverting the volume size.

1. Update the following fields in your values.yaml file based on the volumes you resized:

PersistentVolumeClaims Name	Helm Value
brokerdata-broker-*	broker.persistence.data.size
brokerlog-broker-*	broker.persistence.log.size

1. Delete the StatefulSet while leaving the pods as orphans. This allows Helm to take ownership of the StatefulSet on the next upgrade without conflicting with the manually recreated resource:

kubectl delete sts broker -n ${NAMESPACE} --cascade=orphan

1. Apply the configuration changes by running the helm upgrade command:

helm upgrade -n ${NAMESPACE} ${INSTALLATION_NAME} oci://repo.cybus.io/charts/connectware -f values.yaml

For details, see Applying Helm Configuration Changes.

Deleting Broker Data Volumes

To permanently delete broker data volumes, see Deleting Broker Data Volumes.