Resizing Broker Volumes in Kubernetes

The disk space needed for our brokers depends on customers use-cases, especially in regards to use of QoS > 0 and retained messages, as well as message size. We can therefore not perfectly predict the necessary disk space, allowing for a situation in which existing volumes need to be resized.

This procedure can be used to increase the available disk space for broker volumes with as little service interruption as possible. Currently this means pod restarts, which result in the necessity for clients to reconnect.

This procedure depends on removing the StatefulSet. This leaves the broker cluster open to failures caused by cluster events or human error. Therefore this should be executed with great care and only in a stable cluster.

Prerequisites

1. kubectl access to the necessary installation and the current context namespace set to the target namespace (kubectl config set-context --current --namespace <target-namespace>).

2. A StorageClass that supports volume resizing (kubectl get sc and check that ALLOWVOLUMEEXPANSION show true for the StorageClass used for the volumes.

Instructions

Preparing the Broker Cluster

1. Make sure that you have a healthy broker cluster of at least two pods (kubectl get sts broker shows READY 2/2 or higher, but same number on both sides of the slash)

2. If you only have a single broker, scale the StatefulSet to two:

kubectl scale sts broker --replicas 2

1. Export the StatefulSet definition to a local file:

kubectl get sts broker -o yaml > broker.yaml

Resizing Volumes

Repeat this part for each broker cluster pod you have.

1. Delete the broker StatufulSet while leaving the pods as orphans:

kubectl delete sts broker --cascade=orphan

1. Set variable $broker to the pod name of the broker you want to resize (e.g. broker-0):

broker=broker-0

1. Delete the broker pod:

kubectl delete pod $broker

1. Increase the PVC size (replace size with the Kubernetes Quantity for the volume, e.g. 5Gi):

kubectl patch pvc brokerdata-$broker --patch '{"spec": { "resources": {"requests": {"storage": "<size>"}}}}'

1. Wait until the PVC shows the correct CAPACITY:

kubectl get pvc brokerdata-$broker

1. Recreate the StatefulSet:

kubectl apply -f broker.yaml

1. Wait for the StatefulSet to recreate the missing pod and wait for the StatefulSet to be ready (kubectl get sts broker shows READY 2/2 or higher, but same number on both sides of the slash)

2. Verify that all cluster members show the same output for the cluster members:

kubectl get pod -lapp=broker -o name | xargs -I % kubectl exec % -- vmq-admin cluster show

• The output should look similar to this:

Defaulted container "broker" out of: broker, wait-for-k8s (init)
+-------------------------------------------------+---------+
| Node                                            | Running |
+-------------------------------------------------+---------+
| [e-mail] | true    |
+-------------------------------------------------+---------+
| [e-mail] | true    |
+-------------------------------------------------+---------+
Defaulted container "broker" out of: broker, wait-for-k8s (init)
+-------------------------------------------------+---------+
| Node                                            | Running |
+-------------------------------------------------+---------+
| [e-mail] | true    |
+-------------------------------------------------+---------+
| [e-mail] | true    |
+-------------------------------------------------+---------+

1. Repeat this step for the next broker, until all volumes are resized

Persisting through Helm values

Once you are done you should adjust your Helm values for Connectware to reflect the changes.

Update the following fields based on what volumes you resized:

PVC Name	Helm Value
brokerdata-broker-*	global.broker.storage.data.size
brokerlog-broker-*	global.broker.storage.log.size