[[ch-kubernetes]]
== LINSTOR Volumes in Kubernetes

indexterm:[Kubernetes]This chapter describes the usage of LINSTOR in Kubernetes
as managed by the operator and with volumes provisioned using the
https://github.com/LINBIT/linstor-csi[LINSTOR CSI plugin].

This Chapter goes into great detail regarding all the install time
options and various configurations possible with LINSTOR and
Kubernetes. For those more interested in a "quick-start" for testing,
or those looking for some examples for reference. We have some
complete <<Helm Install Examples>> of a few common uses near the end
of the chapter.

[[s-kubernetes-overview]]
=== Kubernetes Overview

_Kubernetes_ is a container orchestrator. Kubernetes defines the behavior of
containers and related services via declarative specifications. In this guide,
we'll focus on using `kubectl` to manipulate `.yaml` files that define the
specifications of Kubernetes objects.

[[s-kubernetes-deploy]]
=== Deploying LINSTOR on Kubernetes

[[s-kubernetes-deploy-linstor-operator]]
==== Deploying with the LINSTOR Operator

LINBIT provides a LINSTOR operator to commercial support customers.
The operator eases deployment of LINSTOR on Kubernetes by installing DRBD,
managing Satellite and Controller pods, and other related functions.

The operator itself is installed using a Helm v3 chart as follows:

* Create a kubernetes secret containing your my.linbit.com credentials:
+
----
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-email=<YOUR_EMAIL> --docker-password=<YOUR_PASSWORD>
----
+
The name of this secret must match the one specified in the Helm values,
by default `drbdiocred`.

* Configure the LINSTOR database backend. By default, the chart configures Etcd as database backend. Starting with
  1.7.0, the operator can also configure LINSTOR to use <<s-kubernetes-linstor-k8s-backend,Kubernetes as datastore>>
  directly. If you go the Etcd route, you should configure persistent storage for it:
** Use an existing storage provisioner with a default `StorageClass`.
** <<s-kubernetes-etcd-hostpath-persistence,Use `hostPath` volumes>>.
** Disable persistence, **for basic testing only**. This can be done by adding
   `--set etcd.persistentVolume.enabled=false` to the `helm install` command below.

* Read <<s-kubernetes-storage, the storage guide>> and configure a basic storage setup for LINSTOR

* Read the <<s-kubernetes-securing-deployment,section on securing the deployment>> and configure as needed.

* Select the appropriate kernel module injector using `--set` with the `helm install` command in the final step.

** Choose the injector according to the distribution you are using.
   Select the latest version from one of `drbd9-rhel7`, `drbd9-rhel8`,...  from http://drbd.io/ as appropriate.
   The drbd9-rhel8 image should also be used for RHCOS (OpenShift). For the SUSE CaaS Platform use the SLES injector
   that matches the base system of the CaaS Platform you are using (e.g., `drbd9-sles15sp1`). For example:
+
----
operator.satelliteSet.kernelModuleInjectionImage=drbd.io/drbd9-rhel8:v9.0.30
----

** Only inject modules that are already present on the host machine. If a module is not found, it will be skipped.
+
----
operator.satelliteSet.kernelModuleInjectionMode=DepsOnly
----

** Disable kernel module injection if you are installing DRBD by other means. Deprecated by `DepsOnly`
+
----
operator.satelliteSet.kernelModuleInjectionMode=None
----

* Finally create a Helm deployment named `linstor-op` that will set up everything.
+
----
helm repo add linstor https://charts.linstor.io
helm install linstor-op linstor/linstor
----
Further deployment customization is discussed in the <<s-kubernetes-advanced-deployments,advanced deployment section>>

[[s-kubernetes-linstor-k8s-backend]]
===== LINSTOR k8s backend

WARNING: The Kubernetes backend for LINSTOR is still new, you might encounter issues, especially on LINSTOR version
upgrade. Please read <<s-kubernetes-upgrade-k8s-backend, the upgrade guide>> for more information.

Starting with LINSTOR 1.16.0, the Controller can use the Kubernetes API directly to persist its cluster state. To enable
this backend, use the following override file during the chart installation:

.k8s-backend.yaml
[source,yaml]
----
etcd:
  enabled: false
operator:
  controller:
    dbConnectionURL: k8s
----

NOTE: It is currently **NOT** possible to migrate from an existing cluster with Etcd backend to the Kubernetes
backend.

[[s-kubernetes-etcd-hostpath-persistence]]
===== LINSTOR etcd `hostPath` persistence

You can use the `pv-hostpath` Helm templates to create `hostPath` persistent
volumes. Create as many PVs as needed to satisfy your configured etcd
`replicas` (default 1).

Create the `hostPath` persistent volumes, substituting cluster node
names accordingly in the `nodes=` option:

----
helm repo add linstor https://charts.linstor.io
helm install linstor-etcd linstor/pv-hostpath
----

By default, a PV is created on every `control-plane` node. You can manually select the storage nodes by
passing `--set "nodes={<NODE0>,<NODE1>,<NODE2>}"` to the install command.

[[s-kubernetes-existing-database]]
===== Using an existing database

LINSTOR can connect to an existing PostgreSQL, MariaDB or etcd database. For
instance, for a PostgreSQL instance with the following configuration:

----
POSTGRES_DB: postgresdb
POSTGRES_USER: postgresadmin
POSTGRES_PASSWORD: admin123
----

The Helm chart can be configured to use this database instead of deploying an
etcd cluster by adding the following to the Helm install command:

----
--set etcd.enabled=false --set "operator.controller.dbConnectionURL=jdbc:postgresql://postgres/postgresdb?user=postgresadmin&password=admin123"
----

[[s-kubernetes-storage]]
==== Configuring storage

The LINSTOR operator can automate some basic storage set up for LINSTOR.

===== Configuring storage pool creation

The LINSTOR operator can be used to create LINSTOR storage pools. Creation is under control of the
LinstorSatelliteSet resource:

[source]
----
$ kubectl get LinstorSatelliteSet.linstor.linbit.com linstor-op-ns -o yaml
kind: LinstorSatelliteSet
metadata:
..
spec:
  ..
  storagePools:
    lvmPools:
    - name: lvm-thick
      volumeGroup: drbdpool
    lvmThinPools:
    - name: lvm-thin
      thinVolume: thinpool
      volumeGroup: ""
    zfsPools:
    - name: my-linstor-zpool
      zPool: for-linstor
      thin: true
----

====== At install time

At install time, by setting the value of `operator.satelliteSet.storagePools` when running helm install.

First create a file with the storage configuration like:

[source,yaml]
----
operator:
  satelliteSet:
    storagePools:
      lvmPools:
      - name: lvm-thick
        volumeGroup: drbdpool
----

This file can be passed to the helm installation like this:

[source]
----
helm install -f <file> linstor-op linstor/linstor
----

====== After install

On a cluster with the operator already configured (i.e. after `helm install`),
you can edit the LinstorSatelliteSet configuration like this:

[source]
----
$ kubectl edit LinstorSatelliteSet.linstor.linbit.com <satellitesetname>
----

The storage pool configuration can be updated like in the example above.

===== Preparing physical devices

By default, LINSTOR expects the referenced VolumeGroups, ThinPools and so on to be present. You can use the
`devicePaths: []` option to let LINSTOR automatically prepare devices for the pool. Eligible for automatic configuration
are block devices that:

* Are a root device (no partition)
* do not contain partition information
* have more than 1 GiB

To enable automatic configuration of devices, set the `devicePaths` key on `storagePools` entries:

[source,yaml]
----
  storagePools:
    lvmPools:
    - name: lvm-thick
      volumeGroup: drbdpool
      devicePaths:
      - /dev/vdb
    lvmThinPools:
    - name: lvm-thin
      thinVolume: thinpool
      volumeGroup: linstor_thinpool
      devicePaths:
      - /dev/vdc
      - /dev/vdd
----

Currently, this method supports creation of LVM and LVMTHIN storage pools.

===== `lvmPools` configuration
* `name` name of the LINSTOR storage pool.Required
* `volumeGroup` name of the VG to create.Required
* `devicePaths` devices to configure for this pool.Must be empty and >= 1GiB to be recognized.Optional
* `raidLevel` LVM raid level.Optional
* `vdo` Enable [VDO] (requires VDO tools in the satellite).Optional
* `vdoLogicalSizeKib` Size of the created VG (expected to be bigger than the backing devices by using VDO).Optional
* `vdoSlabSizeKib` Slab size for VDO. Optional

[VDO]: https://www.redhat.com/en/blog/look-vdo-new-linux-compression-layer

===== `lvmThinPools` configuration
* `name` name of the LINSTOR storage pool. Required
* `volumeGroup` VG to use for the thin pool. If you want to use `devicePaths`, you must set this to `""`.
  This is required because LINSTOR does not allow configuration of the VG name when preparing devices.
* `thinVolume` name of the thinpool. Required
* `devicePaths` devices to configure for this pool. Must be empty and >= 1GiB to be recognized. Optional
* `raidLevel` LVM raid level. Optional

NOTE: The volume group created by LINSTOR for LVMTHIN pools will always follow the scheme "linstor_$THINPOOL".

===== `zfsPools` configuration
* `name` name of the LINSTOR storage pool. Required
* `zPool` name of the zpool to use. Must already be present on all machines. Required
* `thin` `true` to use thin provisioning, `false` otherwise. Required

===== Using `automaticStorageType` (DEPRECATED)

_ALL_ eligible devices will be prepared according to the value of `operator.satelliteSet.automaticStorageType`, unless
they are already prepared using the `storagePools` section. Devices are added to a storage pool based on the device
name (i.e. all `/dev/nvme1` devices will be part of the pool `autopool-nvme1`)

The possible values for `operator.satelliteSet.automaticStorageType`:

* `None` no automatic set up (default)
* `LVM` create a LVM (thick) storage pool
* `LVMTHIN` create a LVM thin storage pool
* `ZFS` create a ZFS based storage pool (**UNTESTED**)

[[s-kubernetes-securing-deployment]]
==== Securing deployment
This section describes the different options for enabling security features available when
using this operator. The following guides assume the operator is installed <<s-kubernetes-deploy-linstor-operator,using Helm>>

===== Secure communication with an existing etcd instance

Secure communication to an `etcd` instance can be enabled by providing a CA certificate to the operator in form of a
kubernetes secret. The secret has to contain the key `ca.pem` with the PEM encoded CA certificate as value.

The secret can then be passed to the controller by passing the following argument to `helm install`

----
--set operator.controller.dbCertSecret=<secret name>
----

====== Authentication with `etcd` using certificates

If you want to use TLS certificates to authenticate with an `etcd` database, you need to set the following option on
helm install:

----
--set operator.controller.dbUseClientCert=true
----

If this option is active, the secret specified in the above section must contain two additional keys:

* `client.cert` PEM formatted certificate presented to `etcd` for authentication
* `client.key` private key **in PKCS8 format**, matching the above client certificate.

Keys can be converted into PKCS8 format using `openssl`:

----
openssl pkcs8 -topk8 -nocrypt -in client-key.pem -out client-key.pkcs8
----

===== Configuring secure communication between LINSTOR components

The default communication between LINSTOR components is not secured by TLS. If this is needed for your setup,
follow these steps:

* Create private keys in the java keystore format, one for the controller, one for all satellites:
----
keytool -keyalg rsa -keysize 2048 -genkey -keystore satellite-keys.jks -storepass linstor -alias satellite -dname "CN=XX, OU=satellite, O=Example, L=XX, ST=XX, C=X"
keytool -keyalg rsa -keysize 2048 -genkey -keystore control-keys.jks -storepass linstor -alias control -dname "CN=XX, OU=control, O=Example, L=XX, ST=XX, C=XX"
----
* Create a trust store with the public keys that each component needs to trust:
* Controller needs to trust the satellites
* Nodes need to trust the controller
+
----
keytool -importkeystore -srcstorepass linstor -deststorepass linstor -srckeystore control-keys.jks -destkeystore satellite-trust.jks
keytool -importkeystore -srcstorepass linstor -deststorepass linstor -srckeystore satellite-keys.jks -destkeystore control-trust.jks
----

* Create kubernetes secrets that can be passed to the controller and satellite pods
+
----
kubectl create secret generic control-secret --from-file=keystore.jks=control-keys.jks --from-file=certificates.jks=control-trust.jks
kubectl create secret generic satellite-secret --from-file=keystore.jks=satellite-keys.jks --from-file=certificates.jks=satellite-trust.jks
----
* Pass the names of the created secrets to `helm install`
+
----
--set operator.satelliteSet.sslSecret=satellite-secret --set operator.controller.sslSecret=control-secret
----

IMPORTANT: It is currently **NOT** possible to change the keystore password. LINSTOR expects the passwords to be
`linstor`. This is a current limitation of LINSTOR.

===== Configuring secure communications for the LINSTOR API

Various components need to talk to the LINSTOR controller via its REST interface. This interface can be
secured via HTTPS, which automatically includes authentication. For HTTPS+authentication to work, each component
needs access to:

* A private key
* A certificate based on the key
* A trusted certificate, used to verify that other components are trustworthy

The next sections will guide you through creating all required components.

====== Creating the private keys

Private keys can be created using java's keytool

----
keytool -keyalg rsa -keysize 2048 -genkey -keystore controller.pkcs12 -storetype pkcs12 -storepass linstor -ext san=dns:linstor-op-cs.default.svc -dname "CN=XX, OU=controller, O=Example, L=XX, ST=XX, C=X" -validity 5000
keytool -keyalg rsa -keysize 2048 -genkey -keystore client.pkcs12 -storetype pkcs12 -storepass linstor -dname "CN=XX, OU=client, O=Example, L=XX, ST=XX, C=XX" -validity 5000
----

The clients need private keys and certificate in a different format, so we need to convert it

----
openssl pkcs12 -in client.pkcs12 -passin pass:linstor -out client.cert -clcerts -nokeys
openssl pkcs12 -in client.pkcs12 -passin pass:linstor -out client.key -nocerts -nodes
----

NOTE: The alias specified for the controller key (i.e. `-ext san=dns:linstor-op-cs.default.svc`) has to exactly match the
service name created by the operator. When using `helm`, this is always of the form `<release-name>-cs.<release-namespace>.svc`.

IMPORTANT: It is currently NOT possible to change the keystore password. LINSTOR expects the passwords to be linstor. This is a current limitation of LINSTOR

====== Create the trusted certificates

For the controller to trust the clients, we can use the following command to create a truststore, importing the client certificate

----
keytool -importkeystore -srcstorepass linstor -srckeystore client.pkcs12 -deststorepass linstor -deststoretype pkcs12 -destkeystore controller-trust.pkcs12
----

For the client, we have to convert the controller certificate into a different format

----
openssl pkcs12 -in controller.pkcs12 -passin pass:linstor -out ca.pem -clcerts -nokeys
----

====== Create Kubernetes secrets

Now you can create secrets for the controller and for clients:

----
kubectl create secret generic http-controller --from-file=keystore.jks=controller.pkcs12 --from-file=truststore.jks=controller-trust.pkcs12
kubectl create secret generic http-client --from-file=ca.pem=ca.pem --from-file=client.cert=client.cert --from-file=client.key=client.key
----

The names of the secrets can be passed to `helm install` to configure all clients to use https.

----
--set linstorHttpsControllerSecret=http-controller  --set linstorHttpsClientSecret=http-client
----

===== Automatically set the passphrase for LINSTOR

LINSTOR needs to store confidential data to support encrypted information. This data is protected by a master
passphrase. A passphrase is automatically generated on the first chart install.

If you want to use a custom passphrase, store it in a secret:

----
kubectl create secret generic linstor-pass --from-literal=MASTER_PASSPHRASE=<password>
----

On install, add the following arguments to the helm command:

----
--set operator.controller.luksSecret=linstor-pass
----

===== Helm Install Examples

All the below examples use the following `sp-values.yaml` file. Feel
free to adjust this for your uses and environment. See <<Configuring storage pool creation>>
for further details.

-----
operator:
  satelliteSet:
    storagePools:
      lvmThinPools:
      - name: lvm-thin
        thinVolume: thinpool
        volumeGroup: ""
        devicePaths:
        - /dev/sdb
-----

Default install. Please note this does not setup any persistence for
the backing etcd key-value store.

WARNING: This is not suggested for any use
outside of testing.

-----
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-password=<YOUR_PASSWORD>
helm repo add linstor https://charts.linstor.io
helm install linstor-op linstor/linstor
-----

Install with LINSTOR storage-pools defined at install via
`sp-values.yaml`, persistent hostPath volumes, 3 etcd replicas, and by
compiling the DRBD kernel modules for the host kernels.

This should be adequate for most basic deployments. Please note that
this deployment is not using the pre-compiled DRBD kernel modules just
to make this command more portable. Using the pre-compiled binaries
will make for a much faster install and deployment. Using the
`Compile` option would not be suggested for use in a large Kubernetes clusters.

-----
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-password=<YOUR_PASSWORD>
helm repo add linstor https://charts.linstor.io
helm install linstor-etcd linstor/pv-hostpath --set "nodes={<NODE0>,<NODE1>,<NODE2>}"
helm install -f sp-values.yaml linstor-op linstor/linstor --set etcd.replicas=3 --set operator.satelliteSet.kernelModuleInjectionMode=Compile
-----

Install with LINSTOR storage-pools defined at install via
`sp-values.yaml`, use an already created PostgreSQL DB (preferably
clustered), instead of etcd, and use already compiled kernel modules for
DRBD. Additionally, we'll disable the Stork scheduler in this example.

The PostgreSQL database in this particular example is reachable via a
service endpoint named `postgres`. PostgreSQL itself is configured with
`POSTGRES_DB=postgresdb`, `POSTGRES_USER=postgresadmin`, and
`POSTGRES_PASSWORD=admin123`

-----
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-email=<YOUR_EMAIL> --docker-password=<YOUR_PASSWORD>
helm repo add linstor https://charts.linstor.io
helm install -f sp-values.yaml linstor-op linstor/linstor --set etcd.enabled=false --set "operator.controller.dbConnectionURL=jdbc:postgresql://postgres/postgresdb?user=postgresadmin&password=admin123" --set stork.enabled=false
-----

[[s-kubernetes-helm-terminate]]
===== Terminating Helm deployment

To protect the storage infrastructure of the cluster from accidentally deleting vital components, it is necessary to perform some manual steps before deleting a Helm deployment.

1. Delete all volume claims managed by LINSTOR components. You can use the following command to get a list of volume claims managed by LINSTOR. After checking that none of the listed volumes still hold needed data, you can delete them using the generated kubectl delete command.
+
----
$ kubectl get pvc --all-namespaces -o=jsonpath='{range .items[?(@.metadata.annotations.volume\.beta\.kubernetes\.io/storage-provisioner=="linstor.csi.linbit.com")]}kubectl delete pvc --namespace {.metadata.namespace} {.metadata.name}{"\n"}{end}'
kubectl delete pvc --namespace default data-mysql-0
kubectl delete pvc --namespace default data-mysql-1
kubectl delete pvc --namespace default data-mysql-2
----
+
WARNING: These volumes, once deleted, cannot be recovered.

2. Delete the LINSTOR controller and satellite resources.
+
Deployment of LINSTOR satellite and controller is controlled by the LinstorSatelliteSet and LinstorController resources. You can delete the resources associated with your deployment using kubectl
+
----
kubectl delete linstorcontroller <helm-deploy-name>-cs
kubectl delete linstorsatelliteset <helm-deploy-name>-ns
----
+
After a short wait, the controller and satellite pods should terminate. If they continue to run, you can check the above resources for errors (they are only removed after all associated pods terminate)

3. Delete the Helm deployment.
+
If you removed all PVCs and all LINSTOR pods have terminated, you can uninstall the helm deployment
+
----
helm uninstall linstor-op
----
+
NOTE: Due to the Helm's current policy, the Custom Resource Definitions named LinstorController and LinstorSatelliteSet will not be deleted by the command.
      More information regarding Helm's current position on CRDs can be found
      https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-1-let-helm-do-it-for-you[here].

[[s-kubernetes-advanced-deployments]]
==== Advanced deployment options

The helm charts provide a set of further customization options for advanced use cases.

[source,yaml]
----
global:
  imagePullPolicy: IfNotPresent # empty pull policy means k8s default is used ("always" if tag == ":latest", "ifnotpresent" else) <1>
  setSecurityContext: true # Force non-privileged containers to run as non-root users
# Dependency charts
etcd:
  enabled: true
  persistentVolume:
    enabled: true
    storage: 1Gi
  replicas: 1 # How many instances of etcd will be added to the initial cluster. <2>
  resources: {} # resource requirements for etcd containers <3>
  image:
    repository: gcr.io/etcd-development/etcd
    tag: v3.4.15
stork:
  enabled: false
  storkImage: docker.io/openstorage/stork:2.6.5
  schedulerImage: k8s.gcr.io/kube-scheduler-amd64
  schedulerTag: ""
  replicas: 1 <2>
  storkResources: {} # resources requirements for the stork plugin containers <3>
  schedulerResources: {} # resource requirements for the kube-scheduler containers <3>
  podsecuritycontext: {}
csi:
  enabled: true
  pluginImage: "drbd.io/linstor-csi:v0.17.0"
  csiAttacherImage: k8s.gcr.io/sig-storage/csi-attacher:v3.3.0
  csiLivenessProbeImage: k8s.gcr.io/sig-storage/livenessprobe:v2.5.0
  csiNodeDriverRegistrarImage: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
  csiProvisionerImage: k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
  csiSnapshotterImage: k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1
  csiResizerImage: k8s.gcr.io/sig-storage/csi-resizer:v1.3.0
  controllerReplicas: 1 <2>
  nodeAffinity: {} <4>
  nodeTolerations: [] <4>
  controllerAffinity: {} <4>
  controllerTolerations: [] <4>
  enableTopology: true
  resources: {} <3>
  kubeletPath: /var/lib/kubelet <7>
priorityClassName: ""
drbdRepoCred: drbdiocred
linstorHttpsControllerSecret: "" # <- name of secret containing linstor server certificates+key.
linstorHttpsClientSecret: "" # <- name of secret containing linstor client certificates+key.
controllerEndpoint: "" # <- override to the generated controller endpoint. use if controller is not deployed via operator
psp:
  privilegedRole: ""
  unprivilegedRole: ""
operator:
  replicas: 1 # <- number of replicas for the operator deployment <2>
  image: "drbd.io/linstor-operator:v1.7.0"
  affinity: {} <4>
  tolerations: [] <4>
  resources: {} <3>
  podsecuritycontext: {}
  controller:
    enabled: true
    controllerImage: "drbd.io/linstor-controller:v1.17.0"
    dbConnectionURL: ""
    luksSecret: ""
    dbCertSecret: ""
    dbUseClientCert: false
    sslSecret: ""
    affinity: {} <4>
    tolerations: <4>
      - key: node-role.kubernetes.io/master
        operator: "Exists"
        effect: "NoSchedule"
    resources: {} <3>
    replicas: 1 <2>
    additionalEnv: [] <5>
    additionalProperties: {} <6>
  satelliteSet:
    enabled: true
    satelliteImage: "drbd.io/linstor-satellite:v1.17.0"
    storagePools: {}
    sslSecret: ""
    automaticStorageType: None
    affinity: {} <4>
    tolerations: [] <4>
    resources: {} <3>
    monitoringImage: "drbd.io/drbd-reactor:v0.5.1"
    kernelModuleInjectionImage: "drbd.io/drbd9-rhel7:v9.1.4"
    kernelModuleInjectionMode: ShippedModules
    kernelModuleInjectionResources: {} <3>
    additionalEnv: [] <5>
haController:
  enabled: true
  image: drbd.io/linstor-k8s-ha-controller:v0.2.1
  affinity: {} <4>
  tolerations: [] <4>
  resources: {} <3>
  replicas: 1 <2>
----
<1> Sets the pull policy for all images.
<2> Controls the number of replicas for each component.
<3> Set container resource requests and limits. See https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/[the kubernetes docs].
    Most containers need a minimal amount of resources, except for:
    * `etcd.resources` See the https://etcd.io/docs/v3.4.0/op-guide/hardware/[etcd docs]
    * `operator.controller.resources` Around `700MiB` memory is required
    * `operater.satelliteSet.resources` Around `700MiB` memory is required
    * `operator.satelliteSet.kernelModuleInjectionResources` If kernel modules are compiled, 1GiB of memory is required.
<4> Affinity and toleration determine where pods are scheduled on the cluster. See the
    https://kubernetes.io/docs/concepts/scheduling-eviction/[kubernetes docs on affinity and toleration].
    This may be especially important for the `operator.satelliteSet` and `csi.node*` values. To schedule a pod using
    a LINSTOR persistent volume, the node requires a running LINSTOR satellite and LINSTOR CSI pod.
<5> Sets additional environments variables to pass to the Linstor Controller and Satellites. Uses the same format as
    https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/[the `env` value of a container]
<6> Sets additional properties on the Linstor Controller. Expects a simple mapping of `<property-key>: <value>`.
<7> Kubelet expects every CSI plugin to mount volumes under a specific subdirectory of it's own state directory. By
    default, this state directory is `/var/lib/kubelet`. Some Kubernetes distributions use a different directory:
    * microk8s: `/var/snap/microk8s/common/var/lib/kubelet`

[[s-kubernetes-ha-deployment]]
===== High Availability Deployment
To create a High Availability deployment of all components, take a look at the https://github.com/piraeusdatastore/piraeus-operator/blob/b00fd34/doc/scheduling.md[upstream guide]
The default values are chosen so that scaling the components to multiple replicas ensures that the replicas are placed on different nodes. This ensures
that a single node failures will not interrupt the service.

[[s-kubernetes-monitoring]]
===== Monitoring with Prometheus
Starting with Linstor Operator v1.5.0, you can use https://prometheus.io/[Prometheus] to monitor Linstor components.
The operator will set up monitoring containers along the existing components and make them available as a `Service`.

If you use the https://prometheus-operator.dev/[Prometheus Operator], the Linstor Operator will also set up the `ServiceMonitor`
instances. The metrics will automatically be collected by the Prometheus instance associated to the operator, assuming
https://prometheus-operator.dev/docs/kube/monitoring-other-namespaces/[watching the Piraeus namespace is enabled].

To disable exporting of metrics, set `operator.satelliteSet.monitoringImage` to an empty value.

====== Linstor Controller Monitoring

The Linstor Controller exports cluster-wide metrics. Metrics are exported on the existing controller service, using the
path https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-linstor-monitoring[`/metrics`].

====== DRBD Resource Monitoring

All satellites are bundled with a secondary container that uses https://github.com/LINBIT/drbd-reactor/[`drbd-reactor`]
to export metrics directly from DRBD. The metrics are available on port 9942, for convenience a headless service named
`<linstorsatelliteset-name>-monitoring` is provided.

If you want to disable the monitoring container, set `monitoringImage` to `""` in your LinstorSatelliteSet resource.

[[s-kubernetes-deploy-external-controller]]
==== Deploying with an external LINSTOR controller

The operator can configure the satellites and CSI plugin to use an existing LINSTOR setup. This can be useful in cases
where the storage infrastructure is separate from the Kubernetes cluster. Volumes can be provisioned in diskless mode
on the Kubernetes nodes while the storage nodes will provide the backing disk storage.

To skip the creation of a LINSTOR Controller deployment and configure the other components to use your existing LINSTOR
Controller, use the following options when running `helm install`:

* `operator.controller.enabled=false` This disables creation of the `LinstorController` resource
* `operator.etcd.enabled=false` Since no LINSTOR Controller will run on Kubernetes, no database is required.
* `controllerEndpoint=<url-of-linstor-controller>` The HTTP endpoint of the existing LINSTOR Controller. For example: `http://linstor.storage.cluster:3370/`

After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup.

IMPORTANT: Your kubernetes nodes must be reachable using their IP by the controller and storage nodes.

Create a storage class referencing an existing storage pool on your storage nodes.

[source,yaml]
----
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-on-k8s
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: "3"
  storagePool: existing-storage-pool
  resourceGroup: linstor-on-k8s
----

You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes
with the given storage pool, i.e. your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI
will create a diskless resource on the Kubernetes node and attach over the network to the diskful resource.

[[s-kubernetes-deploy-piraeus-operator]]
==== Deploying with the Piraeus Operator

The community supported edition of the LINSTOR deployment in Kubernetes is
called Piraeus. The Piraeus project provides
https://github.com/piraeusdatastore/piraeus-operator[an operator] for
deployment.

[[s-kubernetes-linstor-interacting]]
=== Interacting with LINSTOR in Kubernetes

The Controller pod includes a LINSTOR Client, making it easy to interact directly with LINSTOR.
For instance:

----
kubectl exec deployment/linstor-op-cs-controller -- linstor storage-pool list
----

For a convenient shortcut to the above command, download https://github.com/piraeusdatastore/kubectl-linstor/releases[`kubectl-linstor`]
and install it alongside `kubectl`. Then you can use `kubectl linstor` to get access to the complete Linstor CLI

----
$ kubectl linstor node list
╭───────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node                                      ┊ NodeType   ┊ Addresses                   ┊ State  ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════╡
┊ kube-node-01.test                         ┊ SATELLITE  ┊ 10.43.224.26:3366 (PLAIN)   ┊ Online ┊
┊ kube-node-02.test                         ┊ SATELLITE  ┊ 10.43.224.27:3366 (PLAIN)   ┊ Online ┊
┊ kube-node-03.test                         ┊ SATELLITE  ┊ 10.43.224.28:3366 (PLAIN)   ┊ Online ┊
┊ linstor-op-cs-controller-85b4f757f5-kxdvn ┊ CONTROLLER ┊ 172.24.116.114:3366 (PLAIN) ┊ Online ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────╯
----

It also expands references to PVCs to the matching Linstor resource

----
$ kubectl linstor resource list -r pvc:my-namespace/demo-pvc-1 --all
pvc:my-namespace/demo-pvc-1 -> pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node              ┊ Port ┊ Usage  ┊ Conns ┊    State   ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526 ┊ kube-node-01.test ┊ 7000 ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2021-02-05 09:16:09 ┊
┊ pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526 ┊ kube-node-02.test ┊ 7000 ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2021-02-05 09:16:08 ┊
┊ pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526 ┊ kube-node-03.test ┊ 7000 ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2021-02-05 09:16:09 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
----

It also expands references of the form `pod:[<namespace>/]<podname>` into a list resources in use by the pod.

This should only be necessary for investigating problems and accessing advanced functionality.
Regular operation such as creating volumes should be achieved via the
<<s-kubernetes-basic-configuration-and-deployment,Kubernetes integration>>.

[[s-kubernetes-basic-configuration-and-deployment]]
=== Basic Configuration and Deployment

Once all linstor-csi __Pod__s are up and running, we can provision volumes
using the usual Kubernetes workflows.

Configuring the behavior and properties of LINSTOR volumes deployed via Kubernetes
is accomplished via the use of __StorageClass__es.

IMPORTANT: the "resourceGroup" parameter is mandatory. Usually you want it to be unique and the same as the storage class name.

Here below is the simplest practical _StorageClass_ that can be used to deploy volumes:

.linstor-basic-sc.yaml
[source,yaml]
----
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  # The name used to identify this StorageClass.
  name: linstor-basic-storage-class
  # The name used to match this StorageClass with a provisioner.
  # linstor.csi.linbit.com is the name that the LINSTOR CSI plugin uses to identify itself
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  # LINSTOR will provision volumes from the drbdpool storage pool configured
  # On the satellite nodes in the LINSTOR cluster specified in the plugin's deployment
  storagePool: "drbdpool"
  resourceGroup: "linstor-basic-storage-class"
  # Setting a fstype is required for "fsGroup" permissions to work correctly.
  # Currently supported: xfs/ext4
  csi.storage.k8s.io/fstype: xfs
----

We can create the _StorageClass_ with the following command:

----
kubectl create -f linstor-basic-sc.yaml
----

Now that our _StorageClass_ is created, we can now create a _PersistentVolumeClaim_
which can be used to provision volumes known both to Kubernetes and LINSTOR:

.my-first-linstor-volume-pvc.yaml
[source,yaml]
----
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-first-linstor-volume
spec:
  storageClassName: linstor-basic-storage-class
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
----

We can create the _PersistentVolumeClaim_ with the following command:

----
kubectl create -f my-first-linstor-volume-pvc.yaml
----

This will create a _PersistentVolumeClaim_, but no volume will be created just yet.
The storage class we used specified `volumeBindingMode: WaitForFirstConsumer`, which
means that the volume is only created once a workload starts using it. This ensures
that the volume is placed on the same node as the workload.

For our example, we create a simple Pod, which mounts or volume by referencing the
_PersistentVolumeClaim_.
.my-first-linstor-volume-pod.yaml
[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
  name: fedora
  namespace: default
spec:
  containers:
  - name: fedora
    image: fedora
    command: [/bin/bash]
    args: ["-c", "while true; do sleep 10; done"]
    volumeMounts:
    - name: my-first-linstor-volume
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: my-first-linstor-volume
    persistentVolumeClaim:
      claimName: "my-first-linstor-volume"
----

We can create the _Pod_ with the following command:

----
kubectl create -f my-first-linstor-volume-pod.yaml
----

Running `kubectl describe pod fedora` can be used to confirm that _Pod_
scheduling and volume attachment succeeded. Taking a look at the _PersistentVolumeClaim_,
we can see that it is now bound to a volume.

To remove a volume, please ensure that no pod is using it and then delete the
_PersistentVolumeClaim_ via `kubectl`. For example, to remove the volume that we
just made, run the following two commands, noting that the _Pod_ must be
unscheduled before the _PersistentVolumeClaim_ will be removed:

----
kubectl delete pod fedora # unschedule the pod.

kubectl get pod -w # wait for pod to be unscheduled

kubectl delete pvc my-first-linstor-volume # remove the PersistentVolumeClaim, the PersistentVolume, and the LINSTOR Volume.
----

[[s-kubernetes-sc-parameters]]
==== Available parameters in a StorageClass

The following storage class contains all currently available parameters to configure the provisioned storage.

NOTE: `linstor.csi.linbit.com/` is an optional, but recommended prefix for LINSTOR CSI specific parameters.

[source,yaml]
----
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: full-example
provisioner: linstor.csi.linbit.com
parameters:
  # CSI related parameters
  csi.storage.k8s.io/fstype: xfs
  # LINSTOR parameters
  linstor.csi.linbit.com/autoPlace: "2"
  linstor.csi.linbit.com/placementCount: "2"
  linstor.csi.linbit.com/resourceGroup: "full-example"
  linstor.csi.linbit.com/storagePool: "my-storage-pool"
  linstor.csi.linbit.com/disklessStoragePool: "DfltDisklessStorPool"
  linstor.csi.linbit.com/layerList: "drbd storage"
  linstor.csi.linbit.com/placementPolicy: "AutoPlaceTopology"
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
  linstor.csi.linbit.com/encryption: "true"
  linstor.csi.linbit.com/nodeList: "diskful-a diskful-b"
  linstor.csi.linbit.com/clientList: "diskless-a diskless-b"
  linstor.csi.linbit.com/replicasOnSame: "zone=a"
  linstor.csi.linbit.com/replicasOnDifferent: "rack"
  linstor.csi.linbit.com/disklessOnRemaining: "false"
  linstor.csi.linbit.com/doNotPlaceWithRegex: "tainted.*"
  linstor.csi.linbit.com/fsOpts: "-E nodiscard"
  linstor.csi.linbit.com/mountOpts: "noatime"
  linstor.csi.linbit.com/postMountXfsOpts: "extsize 2m"
  # Linstor properties
  property.linstor.csi.linbit.com/*: <x>
  # DRBD parameters
  DrbdOptions/*: <x>
----

[[s-kubernetes-filesystem]]
==== csi.storage.k8s.io/fstype

Sets the file system type to create for `volumeMode: FileSystem` PVCs. Currently supported are:

* `ext4` (default)
* `xfs`

[[s-kubernetes-autoplace]]
==== autoPlace

`autoPlace` is an integer that determines the amount of replicas a volume of
this _StorageClass_ will have.  For instance, `autoPlace: "3"` will produce
volumes with three-way replication. If neither `autoPlace` nor `nodeList` are
set, volumes will be <<s-autoplace-linstor,automatically placed>> on one node.

IMPORTANT: If you use this option, you must not use <<s-kubernetes-nodelist,nodeList>>.

IMPORTANT: You have to use quotes, otherwise Kubernetes will complain about a malformed _StorageClass_.

TIP: This option (and all options which affect autoplacement behavior) modifies the
number of LINSTOR nodes on which the underlying storage for volumes will be
provisioned and is orthogonal to which _kubelets_ those volumes will be accessible
from.

==== placementCount

`placementCount` is an alias for <<s-kubernetes-autoplace,`autoPlace`>>

==== resourceGroup

The <<s-linstor-resource-groups, LINSTOR Resource Group (RG)>> to associate with this StorageClass. If not set,
a new RG will be created for each new PVC.

[[s-kubernetes-storagepool]]
==== storagePool

`storagePool` is the name of the LINSTOR <<s-storage_pools,storage pool>> that
will be used to provide storage to the newly-created volumes.

CAUTION: Only nodes configured with this same _storage pool_ with be considered
for <<s-kubernetes-autoplace,autoplacement>>. Likewise, for _StorageClasses_ using
<<s-kubernetes-nodelist,nodeList>> all nodes specified in that list must have this
_storage pool_ configured on them.

[[s-kubernetes-disklessstoragepool]]
==== disklessStoragePool

`disklessStoragePool` is an optional parameter that only effects LINSTOR volumes
assigned disklessly to _kubelets_ i.e., as clients. If you have a custom
_diskless storage pool_ defined in LINSTOR, you'll specify that here.

==== layerList

A comma-separated list of layers to use for the created volumes. The available layers and their order are described
towards the end of <<s-linstor-without-drbd, this section>>. Defaults to `drbd,storage`

[[s-kubernetes-placementpolicy]]
==== placementPolicy

Select from one of the available volume schedulers:

* `AutoPlaceTopology`, the default: Use topology information from Kubernetes together with user provided constraints (see <<s-kubernetes-replicasonsame>> and <<s-kubernetes-replicasondifferent>>).
* `AutoPlace` Use LINSTOR autoplace, influenced by <<s-kubernetes-replicasonsame>> and <<s-kubernetes-replicasondifferent>>
* `FollowTopology`: Use CSI Topology information to place at least one volume in each "preferred" zone. Only useable if CSI Topology is enabled.
* `Manual`: Use only the nodes listed in `nodeList` and `clientList`.
* `Balanced`: **EXPERIMENTAL** Place volumes across failure domains, using the least used storage pool on each selected node.

[[s-kubernetes-params-allow-remote-volume-access]]
==== allowRemoteVolumeAccess

Control on which nodes a volume is accessible. The value for this option can take two different forms:

- A simple `"true"` or `"false"` allows access from all nodes, or only those nodes with diskfull resources.

- Advanced rules, which allow more granular rules on which nodes can access the volume.
+
The current implementation can grant access to the volume for nodes that share the same labels. For example, if you want
to allow access from all nodes in the same region and zone as a diskfull resource, you could use:
+
[source,yaml]
----
parameters:
  linstor.csi.linbit.com/allowRemoteVolumeAccess: |
    - fromSame:
      - topology.kubernetes.io/region
      - topology.kubernetes.io/zone
----
+
You can specify multiple rules. The rules are additive, a node only need to match one rule to be assignable.

[[s-kubernetes-encryption]]
==== encryption

`encryption` is an optional parameter that determines whether to encrypt
volumes. LINSTOR must be <<s-linstor-encrypted-volumes,configured for encryption>>
for this to work properly.

[[s-kubernetes-nodelist]]
==== nodeList

`nodeList` is a list of nodes for volumes to be assigned to. This will assign
the volume to each node and it will be replicated among all of them. This
can also be used to select a single node by hostname, but it's more flexible to use
<<s-kubernetes-replicasonsame,replicasOnSame>> to select a single node.

IMPORTANT: If you use this option, you must not use <<s-kubernetes-autoplace,autoPlace>>.

TIP: This option determines on which LINSTOR nodes the underlying storage for volumes
will be provisioned and is orthogonal from which _kubelets_ these volumes will be
accessible.

==== clientList

`clientList` is a list of nodes for diskless volumes to be assigned to. Use in conjunction with <<s-kubernetes-nodelist>>.

[[s-kubernetes-replicasonsame]]
==== replicasOnSame

// These should link to the linstor documentation about node properties, but those
// do not exist at the time of this commit.
`replicasOnSame` is a list of `key` or `key=value` items used as autoplacement selection
labels when <<s-kubernetes-autoplace,autoplace>> is used to determine where to
provision storage. These labels correspond to LINSTOR node properties.

NOTE: The operator periodically synchronizes all labels from Kubernetes Nodes, so you can use them as keys for
scheduling constraints.

Let's explore this behavior with examples assuming a LINSTOR cluster such that `node-a` is configured with the
following auxiliary property `zone=z1` and `role=backups`, while `node-b` is configured with
only `zone=z1`.

If we configure a _StorageClass_ with `autoPlace: "1"` and `replicasOnSame: "zone=z1 role=backups"`,
then all volumes created from that _StorageClass_ will be provisioned on `node-a`,
since that is the only node with all of the correct key=value pairs in the LINSTOR
cluster. This is the most flexible way to select a single node for provisioning.

IMPORTANT: This guide assumes LINSTOR CSI version 0.10.0 or newer. All properties referenced in `replicasOnSame`
and `replicasOnDifferent` are interpreted as auxiliary properties. If you are using an older version of LINSTOR CSI, you
need to add the `Aux/` prefix to all property names. So `replicasOnSame: "zone=z1"` would be `replicasOnSame: "Aux/zone=z1"`
Using `Aux/` manually will continue to work on newer LINSTOR CSI versions.

If we configure a _StorageClass_ with `autoPlace: "1"` and `replicasOnSame: "zone=z1"`,
then volumes will be provisioned on either `node-a` or `node-b` as they both have
the `zone=z1` aux prop.

If we configure a _StorageClass_ with `autoPlace: "2"` and `replicasOnSame: "zone=z1 role=backups"`,
then provisioning will fail, as there are not two or more nodes that have
the appropriate auxiliary properties.

If we configure a _StorageClass_ with `autoPlace: "2"` and `replicasOnSame: "zone=z1"`,
then volumes will be provisioned on both `node-a` and `node-b` as they both have
the `zone=z1` aux prop.

You can also use a property key without providing a value to ensure all replicas are placed on nodes with the same property value,
with caring about the particular value. Assuming there are 4 nodes, `node-a1` and `node-a2` are configured with `zone=a`. `node-b1` and `node-b2`
are configured with `zone=b`. Using `autoPlace: "2"` and `replicasOnSame: "zone"` will place on either `node-a1` and `node-a2` OR on `node-b1` and `node-b2`.

[[s-kubernetes-replicasondifferent]]
==== replicasOnDifferent

`replicasOnDifferent` takes a list of properties to consider, same as <<s-kubernetes-replicasonsame,replicasOnSame>>.
There are two modes of using `replicasOnDifferent`:

* Preventing volume placement on specific nodes:
+
If a value is given for the property, the nodes which have that property-value pair assigned will be considered last.
+
Example: `replicasOnDifferent: "no-csi-volumes=true"` will place no volume on any node with property
`no-csi-volumes=true` unless there are not enough other nodes to fulfill the `autoPlace` setting.

* Distribute volumes across nodes with different values for the same key:
+
If no property value is given, LINSTOR will place the volumes across nodes with different values for that property if
possible.
+
Example: Assuming there are 4 nodes, `node-a1` and `node-a2` are configured with `zone=a`. `node-b1` and `node-b2`
are configured with `zone=b`. Using a _StorageClass_ with `autoPlace: "2"` and `replicasOnDifferent: "zone"`,
LINSTOR will create one replica on either `node-a1` or `node-a2` _and_ one replica on either `node-b1` or `node-b2`.

==== disklessOnRemaining

Create a diskless resource on _all_ nodes that were not assigned a diskful resource.

==== doNotPlaceWithRegex

Do not place the resource on a node which has a resource with a name matching the regex.

[[s-kubernetes-fsops]]
==== fsOpts
`fsOpts` is an optional parameter that passes options to the volume's
filesystem at creation time.

IMPORTANT: Please note these values are specific to your chosen
<<s-kubernetes-filesystem, filesystem>>.

[[s-kubernetes-mountops]]
==== mountOpts
`mountOpts` is an optional parameter that passes options to the volume's
filesystem at mount time.

==== postMountXfsOpts

Extra arguments to pass to `xfs_io`, which gets called before right before first use of the volume.

[[s-kubernetes-storage-class-properties]]
==== property.linstor.csi.linbit.com/*

Parameters starting with `property.linstor.csi.linbit.com/` are translated to Linstor properties that are set on the
<<s-linstor-resource-groups,Resource Group>> associated with the StorageClass.

For example, to set `DrbdOptions/auto-quorum` to `disabled`, use:

----
property.linstor.csi.linbit.com/DrbdOptions/auto-quorum: disabled
----

The full list of options is available https://app.swaggerhub.com/apis-docs/Linstor/Linstor/1.7.0#/developers/resourceDefinitionModify[here]

====  DrbdOptions/*: <x>

NOTE: This option is deprecated, use the more general <<s-kubernetes-storage-class-properties, `property.linstor.csi.linbit.com/*`>> form.

Advanced DRBD options to pass to LINSTOR. For example, to change the replication protocol, use
`DrbdOptions/Net/protocol: "A"`.

[[s-kubernetes-snapshots]]
=== Snapshots

Creating <<s-linstor-snapshots, snapshots>> and creating new volumes from
snapshots is done via the use of __VolumeSnapshot__s, __VolumeSnapshotClass__es,
and __PVC__s.

[[s-kubernetes-add-snaphot-support]]
==== Adding snapshot support

LINSTOR supports the volume snapshot feature, which is configured in some, but not all Kubernetes distributions.

To check if your Kubernetes distribution supports snapshots out of the box, run the following command:

----
$ kubectl get --raw /apis/snapshot.storage.k8s.io/v1
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"snapshot.storage.k8s.io/v1"...
$ # If your distribution does NOT support snapshots out of the box, you will get a different response:
$ kubectl get --raw /apis/snapshot.storage.k8s.io/v1
Error from server (NotFound): the server could not find the requested resource
----

:snapshot-controller-link: https://github.com/kubernetes-csi/external-snapshotter/
:piraeus-charts-link: https://artifacthub.io/packages/helm/piraeus-charts/snapshot-controller
:piraeus-org: https://github.com/piraeusdatastore

In case your Kubernetes distribution _does not_ support snapshots, you can manually add the
{snapshot-controller-link}[required components] from the Kubernetes Storage SIG. For convenience, you can use
{piraeus-charts-link}[Helm Charts] provided by the {piraeus-org}[Piraeus team] to add these components.

.Adding snapshot support using the Piraeus Charts
----
$ kubectl create namespace snapshot-controller
$ helm repo add piraeus-charts https://piraeus.io/helm-charts/
$ helm install -n snapshot-controller snapshot-validation-webhook piraeus-charts/snapshot-validation-webhook
$ helm install -n snapshot-controller snapshot-controller piraeus-charts/snapshot-controller --wait
----

[[s-kubernetes-use-snapshot]]
==== Using volume snapshots
Then we can create our _VolumeSnapshotClass_:

.my-first-linstor-snapshot-class.yaml
[source,yaml]
----
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: my-first-linstor-snapshot-class
driver: linstor.csi.linbit.com
deletionPolicy: Delete
----

Create the _VolumeSnapshotClass_ with `kubectl`:

----
kubectl create -f my-first-linstor-snapshot-class.yaml
----

Now we will create a volume snapshot for the volume that we created above. This
is done with a _VolumeSnapshot_:

.my-first-linstor-snapshot.yaml
[source,yaml]
----
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: my-first-linstor-snapshot
spec:
  volumeSnapshotClassName: my-first-linstor-snapshot-class
  source:
    persistentVolumeClaimName: my-first-linstor-volume
----

Create the _VolumeSnapshot_ with `kubectl`:

----
kubectl create -f my-first-linstor-snapshot.yaml
----

You can check that the snapshot creation was successful

----
kubectl describe volumesnapshots.snapshot.storage.k8s.io my-first-linstor-snapshot
...
Spec:
  Source:
    Persistent Volume Claim Name:  my-first-linstor-snapshot
  Volume Snapshot Class Name:      my-first-linstor-snapshot-class
Status:
  Bound Volume Snapshot Content Name:  snapcontent-b6072ab7-6ddf-482b-a4e3-693088136d2c
  Creation Time:                       2020-06-04T13:02:28Z
  Ready To Use:                        true
  Restore Size:                        500Mi
----

Finally, we'll create a new volume from the snapshot with a _PVC_.

.my-first-linstor-volume-from-snapshot.yaml
[source,yaml]
----
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-first-linstor-volume-from-snapshot
spec:
  storageClassName: linstor-basic-storage-class
  dataSource:
    name: my-first-linstor-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
----

Create the _PVC_ with `kubectl`:

----
kubectl create -f my-first-linstor-volume-from-snapshot.yaml
----


[[s-kubernetes-volume-accessibility-and-locality]]
=== Volume Accessibility and Locality
// This only covers DRBD volumes, section might change if linked docs are updated.
LINSTOR volumes are typically accessible both locally and <<s-drbd_clients,over the network>>. The CSI driver will
ensure that the volume is accessible on whatever node was selected for the consumer. The driver also provides options
to ensure volume locality (the consumer is placed on the same node as the backing data) and restrict accessibility
(only a subset of nodes can access the volume over the network).

Volume locality is achieved by setting `volumeBindingMode: WaitForFirstConsumer` in the storage class. This tell
Kubernetes and the CSI driver to wait until the first consumer (Pod) referencing the PVC is scheduled. The CSI driver
then provisions the volume with backing data on the same node as the consumer. In case a node without appropriate
storage pool was selected, a replacement node in the set of accessible nodes is chosen (see below).

Volume accessibility is controlled by the
<<s-kubernetes-params-allow-remote-volume-access,`allowRemoteVolumeAccess` parameter>>. Whenever the CSI plugin needs to
place a volume, this parameter is consulted to get the set of "accessible" nodes. This means they can share volumes
placed on them via the network. This information is also propagated to Kubernetes via label selectors on the PV.

==== Volume Accessibility and Locality examples

The following example show common scenarios where you want to optimize volume accessibility and locality. It also
includes examples of how to spread volume replicas across zones in a cluster.

===== Single-zone homogeneous cluster

The cluster only spans a single zone, i.e. latency between nodes is low. The cluster is homogeneous, i.e. all nodes
are configured similarly, all nodes have their own local storage pool.

.example-storage-class.yaml
[source,yaml]
----
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer <1>
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool <2>
  linstor.csi.linbit.com/placementCount: "2" <3>
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" <4>
----

<1> Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
<2> Set the storage pool(s) to use.
<3> Ensure that the data is replicated, so that at least 2 nodes store the data.
<4> Allow using the volume even on nodes without replica. Since all nodes are connected equally, performance impact
    should be manageable.

=== Multi-zone homogenous cluster

As before, in our homogenous cluster all nodes are configured similarly with their own local storage pool. The cluster
spans now multiple zones, with increased latency across nodes in different zones. To ensure low latency, we want
to restrict access to the volume with a local replica to only those zones that do have a replica. At the same time,
we want to spread our data across multiple zones.

.example-storage-class.yaml
[source,yaml]
----
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer <1>
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool <2>
  linstor.csi.linbit.com/placementCount: "2" <3>
  linstor.csi.linbit.com/allowRemoteVolumeAccess: | <4>
    - fromSame:
      - topology.kubernetes.io/zone
  linstor.csi.linbit.com/replicasOnDifferent: topology.kubernetes.io/zone <5>
----

<1> Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
<2> Set the storage pool(s) to use.
<3> Ensure that the data is replicated, so that at least 2 nodes store the data.
<4> Allow using the volume on nodes in the same zone as a replica, under the assumption that zone internal networking
    is fast and low latency.
<5> Spread the replicas across different zones.

=== Multi-region cluster

Our cluster now spans multiple regions. We don't want to incur the latency penalty to replica our data across regions,
we just want to replica in the same zone.

.example-storage-class.yaml
[source,yaml]
----
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer <1>
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool <2>
  linstor.csi.linbit.com/placementCount: "2" <3>
  linstor.csi.linbit.com/allowRemoteVolumeAccess: | <4>
    - fromSame:
      - topology.kubernetes.io/zone
  linstor.csi.linbit.com/replicasOnSame: topology.kubernetes.io/region <5>
----

<1> Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
<2> Set the storage pool(s) to use.
<3> Ensure that the data is replicated, so that at least 2 nodes store the data.
<4> Allow using the volume on nodes in the same zone as a replica, under the assumption that zone internal networking
    is fast and low latency.
<5> Restrict replicas to only a single region.

=== Cluster with external storage

Our cluster is now only comprised of compute nodes without local storage. Any volume access has to occur via remote
volume access.

.example-storage-class.yaml
[source,yaml]
----
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool <1>
  linstor.csi.linbit.com/placementCount: "1" <2>
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" <3>
----

<1> Set the storage pool(s) to use.
<2> Assuming we only have one storage host, we can only place a single volume without additional replicas
<3> Our worker nodes need to be allowed to connect to the external storage host


[[s-kubernetes-stork]]
=== Volume Locality Optimization using Stork

WARNING: The Stork integration is no longer actively maintained. Instead we recommend using
`volumeBindingMode: WaitForFirstConsumer` in your storage class. For more details
<<s-kubernetes-volume-accessibility-and-locality,see above>>.

Stork is a scheduler extender plugin for Kubernetes which allows a storage
driver to give the Kubernetes scheduler hints about where to place a new pod
so that it is optimally located for storage performance. You can learn more
about the project on its https://portworx.com/stork-storage-orchestration-kubernetes/[GitHub page].

The next Stork release will include the LINSTOR driver by default.
In the meantime, you can use a custom-built Stork container by LINBIT which includes a LINSTOR driver,
https://hub.docker.com/repository/docker/linbit/stork[available on Docker Hub]

[[s-kubernetes-using-stork]]
==== Using Stork

By default, the chart will not install Stork. To install it and register a new scheduler called `stork`
with Kubernetes, pass `--set stork.enabled=false` to the install command.
The new scheduler can be used to place pods near to their volumes.

[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  schedulerName: stork <1>
  containers:
  - name: busybox
    image: busybox
    command: ["tail", "-f", "/dev/null"]
    volumeMounts:
    - name: my-first-linstor-volume
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: my-first-linstor-volume
    persistentVolumeClaim:
      claimName: "test-volume"
----

<1> Add the name of the scheduler to your pod.

[[s-kubernetes-ha-controller]]
=== Fast workload fail over using the High Availability Controller

The LINSTOR High Availability Controller (HA Controller) will speed up the fail over process for stateful workloads using LINSTOR for storage.
It is deployed by default, and can be scaled to multiple replicas:

[source]
----
$ kubectl get pods -l app.kubernetes.io/name=linstor-op-ha-controller
NAME                                       READY   STATUS    RESTARTS   AGE
linstor-op-ha-controller-f496c5f77-fr76m   1/1     Running   0          89s
linstor-op-ha-controller-f496c5f77-jnqtc   1/1     Running   0          89s
linstor-op-ha-controller-f496c5f77-zcrqg   1/1     Running   0          89s
----

In the event of node failures, Kubernetes is very conservative in rescheduling stateful workloads. This means it can
take more than 15 minutes for Pods to be moved from unreachable nodes. With the information available to DRBD and
LINSTOR, this process can be sped up significantly.

The HA Controller enables fast fail over for:

* Pods using DRBD backed PersistentVolumes. The DRBD resources must make use of the quorum functionality
  LINSTOR will configure this automatically for volumes with 2 or more replicas in clusters with at least 3 nodes.
* The workload does not use any external resources in a way that could lead to a conflicting state if two instances
  try to use the external resource at the same time. While DRBD can ensure that only one instance can have write access
  to the storage, it cannot provide the same guarantee for external resources.
* The Pod is marked with the `linstor.csi.linbit.com/on-storage-lost: remove` label.

==== Example

The following StatefulSet uses the HA Controller to manage fail over of a pod.

[source,yaml]
----
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-stateful-app
spec:
  serviceName: my-stateful-app
  selector:
    matchLabels:
      app.kubernetes.io/name: my-stateful-app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: my-stateful-app
        linstor.csi.linbit.com/on-storage-lost: remove <1>
    ...
----

<1> The label is applied to Pod template, **not** the StatefulSet. The label was applied correctly, if your Pod appears
    in the output of `kubectl get pods -l linstor.csi.linbit.com/on-storage-lost=remove`.

Deploy the set and wait for the pod to start

[source]
----
$ kubectl get pod -o wide
NAME                                        READY   STATUS              RESTARTS   AGE     IP                NODE                    NOMINATED NODE   READINESS GATES
my-stateful-app-0                           1/1     Running             0          5m      172.31.0.1        node01.ha.cluster       <none>           <none>
----

Then one of the nodes becomes unreachable. Shortly after, Kubernetes will mark the node as `NotReady`

[source]
----
$ kubectl get nodes
NAME                    STATUS     ROLES     AGE    VERSION
master01.ha.cluster     Ready      master    12d    v1.19.4
master02.ha.cluster     Ready      master    12d    v1.19.4
master03.ha.cluster     Ready      master    12d    v1.19.4
node01.ha.cluster       NotReady   compute   12d    v1.19.4
node02.ha.cluster       Ready      compute   12d    v1.19.4
node03.ha.cluster       Ready      compute   12d    v1.19.4
----

After about 45 seconds, the Pod will be removed by the HA Controller and re-created by the StatefulSet

----
$ kubectl get pod -o wide
NAME                                        READY   STATUS              RESTARTS   AGE     IP                NODE                    NOMINATED NODE   READINESS GATES
my-stateful-app-0                           0/1     ContainerCreating   0          3s      172.31.0.1        node02.ha.cluster       <none>           <none>
$ kubectl get events --sort-by=.metadata.creationTimestamp -w
...
0s          Warning   ForceDeleted              pod/my-stateful-app-0                                                                   pod deleted because a used volume is marked as failing
0s          Warning   ForceDetached             volumeattachment/csi-d2b994ff19d526ace7059a2d8dea45146552ed078d00ed843ac8a8433c1b5f6f   volume detached because it is marked as failing
...
----
