# Resizing Broker Volumes in Kubernetes

The disk space needed for our brokers depends on customers use-cases, especially in regards to use of QoS > 0 and retained messages, as well as message size. We can therefore not perfectly predict the necessary disk space, allowing for a situation in which existing volumes need to be resized.

This procedure can be used to increase the available disk space for broker volumes with as little service interruption as possible. Currently this means pod restarts, which result in the necessity for clients to reconnect.

{% hint style="warning" %}
This procedure depends on removing the StatefulSet. This leaves the broker cluster open to failures caused by cluster events or human error. Therefore this should be executed with great care and only in a stable cluster.
{% endhint %}

## Prerequisites

1. `kubectl` access to the necessary installation and the current context namespace set to the target namespace (`kubectl config set-context --current --namespace <target-namespace>`).
2. A `StorageClass` that supports volume resizing (`kubectl get sc` and check that `ALLOWVOLUMEEXPANSION` show `true` for the StorageClass used for the volumes.

## Instructions

### Preparing the Broker Cluster

1. Make sure that you have a healthy broker cluster of at least two pods (`kubectl get sts broker` shows `READY 2/2` or higher, but same number on both sides of the slash)
2. If you only have a single broker, scale the StatefulSet to two:

{% code lineNumbers="true" %}

```yaml
kubectl scale sts broker --replicas 2
```

{% endcode %}

3. Export the StatefulSet definition to a local file:

{% code lineNumbers="true" %}

```yaml
kubectl get sts broker -o yaml > broker.yaml
```

{% endcode %}

### Resizing Volumes

Repeat this part for each broker cluster pod you have.

1. Delete the broker StatufulSet while leaving the pods as orphans:

{% code lineNumbers="true" %}

```yaml
kubectl delete sts broker --cascade=orphan
```

{% endcode %}

2. Set variable `$broker` to the pod name of the broker you want to resize (e.g. `broker-0`):

{% code lineNumbers="true" %}

```yaml
broker=broker-0
```

{% endcode %}

3. Delete the broker pod:

{% code lineNumbers="true" %}

```yaml
kubectl delete pod $broker
```

{% endcode %}

4. Increase the PVC size (replace `size` with the Kubernetes Quantity for the volume, e.g. `5Gi`):

{% code lineNumbers="true" %}

```yaml
kubectl patch pvc brokerdata-$broker --patch '{"spec": { "resources": {"requests": {"storage": "<size>"}}}}'
```

{% endcode %}

5. Wait until the PVC shows the correct CAPACITY:

{% code lineNumbers="true" %}

```yaml
kubectl get pvc brokerdata-$broker
```

{% endcode %}

6. Recreate the StatefulSet:

{% code lineNumbers="true" %}

```yaml
kubectl apply -f broker.yaml
```

{% endcode %}

7. Wait for the StatefulSet to recreate the missing pod and wait for the StatefulSet to be ready (`kubectl get sts broker` shows `READY 2/2` or higher, but same number on both sides of the slash)
8. Verify that all cluster members show the same output for the cluster members:

{% code lineNumbers="true" %}

```yaml
kubectl get pod -lapp=broker -o name | xargs -I % kubectl exec % -- vmq-admin cluster show
```

{% endcode %}

* The output should look similar to this:

{% code lineNumbers="true" %}

```yaml
Defaulted container "broker" out of: broker, wait-for-k8s (init)
+-------------------------------------------------+---------+
| Node                                            | Running |
+-------------------------------------------------+---------+
| VerneMQ@broker-0.broker.cybus.svc.cluster.local | true    |
+-------------------------------------------------+---------+
| VerneMQ@broker-1.broker.cybus.svc.cluster.local | true    |
+-------------------------------------------------+---------+
Defaulted container "broker" out of: broker, wait-for-k8s (init)
+-------------------------------------------------+---------+
| Node                                            | Running |
+-------------------------------------------------+---------+
| VerneMQ@broker-0.broker.cybus.svc.cluster.local | true    |
+-------------------------------------------------+---------+
| VerneMQ@broker-1.broker.cybus.svc.cluster.local | true    |
+-------------------------------------------------+---------+
```

{% endcode %}

9. Repeat this step for the next broker, until all volumes are resized

### Persisting through Helm values

Once you are done you should adjust your Helm values for Connectware to reflect the changes.

Update the following fields based on what volumes you resized:

| PVC Name             | Helm Value                        |
| -------------------- | --------------------------------- |
| brokerdata-broker-\* | `global.broker.storage.data.size` |
| brokerlog-broker-\*  | `global.broker.storage.log.size`  |