[[s-kubernetes-upgrade]]
=== Upgrading a LINSTOR Deployment on Kubernetes

A LINSTOR Deployment on Kubernets can be upgraded to a new release using Helm.

Before upgrading to a new release, you should ensure you have an up-to-date backup of the LINSTOR database.
If you are using the Etcd database packaged in the LINSTOR Chart, see <<s-kubernetes-etcd-backup,here>>

IMPORTANT: Upgrades using the LINSTOR Etcd deployment require etcd to use persistent storage. Only follow these steps if
Etcd was deployed using `etcd.persistentVolume.enabled=true`

Upgrades will update to new versions of the following components:

* LINSTOR operator deployment
* LINSTOR Controller
* LINSTOR Satellite
* LINSTOR CSI Driver
* Etcd
* Stork

Some versions require special steps, please take a look <<s-kubernetes-upgrade-version,here>>
The main command to upgrade to a new LINSTOR operator version is:

----
helm repo update
helm upgrade linstor-op linstor/linstor
----

If you used any customizations on the initial install, pass the same options to `helm upgrade`. The options currently
in use can be retrieved from Helm.

----
# Retrieve the currently set options
$ helm get values linstor-op
USER-SUPPLIED VALUES:
USER-SUPPLIED VALUES: null
drbdRepoCred: drbdiocred
operator:
  satelliteSet:
    kernelModuleInjectionImage: drbd.io/drbd9-rhel8:v9.0.28
    storagePools:
      lvmThinPools:
      - devicePaths:
        - /dev/vdb
        name: thinpool
        thinVolume: thinpool
        volumeGroup: ""
# Save current options
$ helm get values linstor-op > orig.yaml
# modify values here as needed. for example selecting a newer DRBD image
$ vim orig.yaml
# Start the upgrade
$ helm upgrade linstor-op linstor/linstor -f orig.yaml
----

This triggers the rollout of new pods. After a short wait, all pods should be running and ready.
Check that no errors are listed in the status section of LinstorControllers, LinstorSatelliteSets and LinstorCSIDrivers.

IMPORTANT: During the upgrade process, provisioning of volumes and attach/detach operations might not work. Existing
volumes and volumes already in use by a pod will continue to work without interruption.

[[s-kubernetes-upgrade-k8s-backend]]
==== Upgrading a LINSTOR Cluster with Kubernetes backend

Using the Kubernetes API as LINSTOR backend is quite new: there might be situations in which LINSTOR fails to upgrade
resources, leaving the LINSTOR cluster in an unusable state. For this reason, we strongly recommend doing a manual
backup of all LINSTOR resources before upgrading the cluster. In a future release, this step will be managed by the
operator.

1. Stop the current controller:
+
----
$ kubectl patch linstorcontroller linstor-cs '{"spec":{"replicas": 0}}'
$ kubectl rollout status --watch deployment/linstor-cs-controller
----
2. The following command will create a file `crds.yaml`, which stores the current state of all LINSTOR Custom Resource
   Definitions:
+
----
$ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | xargs kubectl get crds -oyaml > crds.yaml
----
+
3. In addition to the definitions, the actual resources must be be backed up as well:
+
----
$ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | xargs -i{} sh -c "kubectl get {} -oyaml > {}.yaml"
----
4. Run the chart upgrade using `--set IHaveBackedUpAllMyLinstorResources=true` to acknowledge you have executed the
   above steps.

[[s-kubernetes-upgrade-version]]
==== Upgrade instructions for specific versions

Some versions require special steps, see below.

===== Upgrade to v1.7

No additional steps necessary.

===== Upgrade to v1.6

This versions introduces a new option to support Kubernetes distributions which use different state directories than the
default of `/var/lib/kubelet`. A notable example is microk8s, which uses `/var/snap/microk8s/common/var/lib/kubelet`.
To support this, a small addition to the `LinstorCSIDriver` CRD was necessary. As Helm does not upgrade CRDs on chart
upgrade, instead please run:

----
$ helm repo update
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced
----

If you do not apply the new CRDs, you will get errors like the following:

----
Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(LinstorCSIDriver.spec): unknown field "kubeletPath" in com.linbit.linstor.v1.LinstorCSIDriver.spec
----

If you previously used the included snapshot-controller to process `VolumeSnapshot` resources, you should replace it
with the new charts provided by the Piraeus project. The <<s-kubernetes-add-snaphot-support,section on snapshots>> has
been updated to include instructions on how you can add the snapshot-controller to your cluster.

===== Upgrade to v1.5

This version introduces a <<s-kubernetes-monitoring,monitoring>> component for DRBD resources. This requires a new image
and a replacement of the existing `LinstorSatelliteSet` CRD. Helm does not upgrade the CRDs on a chart upgrade,
instead please run:

----
$ helm repo update
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced
----

If you do not plan to use the provided <<s-kubernetes-monitoring,monitoring>> you still need to apply the above steps,
otherwise you will get an errors like the following

----
Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(LinstorSatelliteSet.spec): unknown field "monitoringImage" in com.linbit.linstor.v1.LinstorSatelliteSet.spec
----

NOTE: Some Helm versions fail to set the monitoring image even after replacing the CRDs. In that case, the in-cluster
LinstorSatelliteSet will show an empty `monitoringImage` value. Edit the resource using
`kubectl edit linstorsatellitesets` and set the value to `drbd.io/drbd-reactor:v0.3.0` to enable monitoring.

===== Upgrade to v1.4

This version introduces a new default version for the Etcd image, so take extra care that Etcd is using
persistent storage. *Upgrading the Etcd image without persistent storage will corrupt the cluster*.

If you are upgrading an existing cluster without making use of new Helm options, no additional steps are necessary.

If you plan to use the newly introduced `additionalProperties` and `additionalEnv` settings, you have to replace
the installed CustomResourceDefinitions with newer versions. Helm does not upgrade the CRDs on a chart upgrade

----
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced
----

===== Upgrade to v1.3

No additional steps necessary.

===== Upgrade to v1.2

LINSTOR operator v1.2 is supported on Kubernetes 1.17+. If you are using an older Kubernetes distribution, you may need
to change the default settings, for example [the CSI provisioner](https://kubernetes-csi.github.io/docs/external-provisioner.html).

There is a known issue when updating the CSI components: the pods will not be updated to the newest image and the
`errors` section of the LinstorCSIDrivers resource shows an error updating the DaemonSet. In this case, manually
delete `deployment/linstor-op-csi-controller` and `daemonset/linstor-op-csi-node`. They will be re-created by the
operator.

[[s-kubernetes-etcd-backup]]
==== Creating Etcd Backups

To create a backup of the Etcd database and store it on your control host, run:

[source]
----
kubectl exec linstor-op-etcd-0 -- etcdctl snapshot save /tmp/save.db
kubectl cp linstor-op-etcd-0:/tmp/save.db save.db
----

These commands will create a file `save.db` on the machine you are running `kubectl` from.
