[[ch-openstack-linstor]]
== LINSTOR Volumes in Openstack

indexterm:[Openstack]indexterm:[Cinder]
This chapter describes using LINSTOR to provision persistent, replicated, and
high-performance block storage for Openstack.


[[s-openstack-linstor-overview]]
=== Openstack Overview

Openstack consists of a wide range of individual services; The service responsible
for provisioning and managing block storage is called
https://docs.openstack.org/cinder/latest/[Cinder]. Other openstack services such
as the compute instance service https://docs.openstack.org/nova/latest[Nova] can
request volumes from Cinder. Cinder will then make a volume accessible to the
requesting service.

LINSTOR can integrate with Cinder using a *volume driver*. The volume driver
translates calls to the Cinder API to LINSTOR commands. For example: requesting a
volume from Cinder will create new resources in LINSTOR, Cinder Volume snapshots
translate to snapshots in LINSTOR and so on.

[[s-openstack-linstor-install]]
=== LINSTOR for Openstack Installation

An initial installation and configuration of DRBD and LINSTOR must be completed
prior to using the OpenStack driver.

* A detailed installation guide can be found <<s-installation,here>>.
* <<s-linstor-init-cluster,Initialize>> your cluster
* Learn how to use the <<s-using_the_linstor_client,LINSTOR client>>
* <<s-adding_nodes_to_your_cluster,Add all storage nodes>> to the LINSTOR cluster

At this point you should be able to list your storage cluster
nodes using the LINSTOR client:

[source]
----
$ linstor node info
╭────────────────────────────────────────────────────────────────────────────╮
┊ Node                      ┊ NodeType  ┊ Addresses                 ┊ State  ┊
╞════════════════════════════════════════════════════════════════════════════╡
┊ cinder-01.openstack.test  ┊ COMBINED  ┊ 10.43.224.21:3366 (PLAIN) ┊ Online ┊
┊ cinder-02.openstack.test  ┊ COMBINED  ┊ 10.43.224.22:3366 (PLAIN) ┊ Online ┊
┊ storage-01.openstack.test ┊ SATELLITE ┊ 10.43.224.11:3366 (PLAIN) ┊ Online ┊
┊ storage-02.openstack.test ┊ SATELLITE ┊ 10.43.224.12:3366 (PLAIN) ┊ Online ┊
┊ storage-03.openstack.test ┊ SATELLITE ┊ 10.43.224.13:3366 (PLAIN) ┊ Online ┊
╰────────────────────────────────────────────────────────────────────────────╯
----

You should <<s-storage_pools,configure one or more storage pools>> per node. This guide assumes the
storage pool is named `cinderpool`. LINSTOR should list the storage pool for each node, including the diskless storage
pool created by default.

[source]
----
$ linstor storage-pool list
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node                      ┊ Driver   ┊ PoolName        ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cinder-01.openstack.test  ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ cinder-02.openstack.test  ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ storage-01.openstack.test ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ storage-02.openstack.test ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ storage-03.openstack.test ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ cinderpool           ┊ storage-01.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊      100 GiB ┊       100 GiB ┊ True         ┊ Ok    ┊
┊ cinderpool           ┊ storage-02.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊      100 GiB ┊       100 GiB ┊ True         ┊ Ok    ┊
┊ cinderpool           ┊ storage-03.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊      100 GiB ┊       100 GiB ┊ True         ┊ Ok    ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
----

[[s-update_the_linstor_driver]]
==== Upgrades of LINSTOR driver

If this is a fresh installation, skip this section and continue with <<s-install_the_linstor_driver>>.

===== From 1.x to 2.x

Driver version 2 dropped some static configuration options in favour of managing these options at runtime via
<<s-openstack-volume-types,volume types>>.

[frame="topbot",options="header"]
|====
| Option in `cinder.conf` | Status | Replace with
| `linstor_autoplace_count` | removed | Use the `linstor:redundancy` property on the volume type. Using a value of 0 for full cluster replication is not supported, use the advanced options of <<s-autoplace-linstor, the LINSTOR autoplacer>>
| `linstor_controller_diskless` | removed | No replacement needed, the driver will create a diskless resource on the cinder host when required
| `linstor_default_blocksize` | removed | This setting had no effect.
| `linstor_default_storage_pool_name` | deprecated | This setting is deprecated for removal in a future version. Use the `linstor:storage_pool` property on the volume type instead.
| `linstor_default_uri` | deprecated | Replaced by the more aptly named `linstor_uris`.
| `linstor_default_volume_group_name` | removed | Creating nodes and storage pools was completely removed in Driver version 2. See <<s-openstack-linstor-install>>
| `linstor_volume_downsize_factor` | removed | This setting served no purpose, it was removed without replacement.
|====

[[s-install_the_linstor_driver]]
==== Install the LINSTOR driver

Starting with OpenStack Stein, the LINSTOR driver is part of the Cinder project. While the driver can be used as is, it
might be missing features or fixes available in newer version. Due to OpenStacks update policy for stable versions, most
improvements to the driver will not get back-ported to older stable releases.

https://github.com/LINBIT/openstack-cinder[LINBIT maintains a fork] of the Cinder repository with all improvements to
the LINSTOR driver backported to the supported stable versions. Currently, these are:

[frame="topbot",options="header"]
|====
| OpenStack Release    | Included Version | LINBIT Version | LINBIT Branch
| `wallaby`            | 1.0.1            | 2.0.0          | https://github.com/LINBIT/openstack-cinder/tree/linstor%2Fstable%2Fwallaby[`linstor/stable/wallaby`]
| `victoria`           | 1.0.1            | 2.0.0          | https://github.com/LINBIT/openstack-cinder/tree/linstor%2Fstable%2Fvictoria[`linstor/stable/victoria`]
| `ussuri`             | 1.0.1            | 2.0.0          | https://github.com/LINBIT/openstack-cinder/tree/linstor%2Fstable%2Fussuri[`linstor/stable/ussuri`]
| `train`              | 1.0.0            | 2.0.0          | https://github.com/LINBIT/openstack-cinder/tree/linstor%2Fstable%2Ftrain[`linstor/stable/train`]
| `stein`              | 1.0.0            | 2.0.0          | https://github.com/LINBIT/openstack-cinder/tree/linstor%2Fstable%2Fstein[`linstor/stable/stein`]
| `rocky`              | n/a              | n/a            | n/a
| `queens`             | n/a              | n/a            | n/a
| `pike`               | n/a              | n/a            | n/a
|====

The exact steps to enable the Linstor Driver depend on your OpenStack distribution. In general, the `python-linstor`
package needs to be installed on all hosts running the Cinder volume service. The next section will cover the
installation process for common OpenStack distributions.

===== DevStack

https://docs.openstack.org/devstack/latest/[DevStack] is a great way to try out OpenStack in a lab environment.
To use the most recent driver use the following DevStack configuration:

.local.conf
----
# This ensures the Linstor Driver has access to the 'python-linstor' package.
#
# This is needed even if using the included driver!
USE_VENV=True
ADDITIONAL_VENV_PACKAGES=python-linstor

# This is required to select the LINBIT version of the driver
CINDER_REPO=https://github.com/LINBIT/openstack-cinder.git
# Replace linstor/stable/victoria with the reference matching your Openstack release.
CINDER_BRANCH=linstor/stable/victoria
----

===== Kolla

https://docs.openstack.org/kolla/latest/[Kolla] packages OpenStack components in containers. They can then be deployed,
for example using https://docs.openstack.org/kolla-ansible/latest/[Kolla Ansible]
You can take advantage of the available customisation options for kolla containers to set up the Linstor driver.

To ensure that the required `python-linstor` package is installed, use the following override file:

.template-override.j2
[source]
----
{% extends parent_template %}

# Cinder
{% set cinder_base_pip_packages_append = ['python-linstor'] %}
----

To install the LINBIT version of the driver, update your `kolla-build.conf`

./etc/kolla/kolla-build.conf
----
[cinder-base]
type = git
location = https://github.com/LINBIT/openstack-cinder.git
# Replace linstor/stable/victoria with the reference matching your Openstack release.
reference = linstor/stable/victoria
----

To rebuild the Cinder containers, run:
[source,shell]
----
# A private registry used to store the kolla container images
REGISTRY=deployment-registry.example.com
# The image namespace in the registry
NAMESPACE=kolla
# The tag to apply to all images. Use the release name for compatibility with kolla-ansible
TAG=victoria
kolla-build -t source --template-override template-override.j2 cinder --registry $REGISTRY --namespace $NAMESPACE --tag $TAG
----

====== Kolla Ansible

When deploying OpenStack using Kolla Ansible, you need to make sure that:

* the custom Cinder images, created in the section above, are used
* deployment of Cinder services is enabled

./etc/kolla/globals.yml
[source,yaml]
----
# use "source" images
kolla_install_type: source
# use the same registry as for running kolla-build above
docker_registry: deployment-registry.example.com
# use the same namespace as for running kolla-build above
docker_namespace: kolla
# deploy cinder block storage service
enable_cinder: "yes"
# disable verification of cinder backends, kolla-ansible only supports a small subset of available backends for this
skip_cinder_backend_check: True
# add the LINSTOR backend to the enabled backends. For backend configuration see below
cinder_enabled_backends:
  - name: linstor-drbd
----

You can place the Linstor driver configuration in one of the override directories for kolla-ansible. For more details on
the available configuration options, see the section below.

./etc/kolla/config/cinder/cinder-volume.conf
----
[linstor-drbd]
volume_backend_name = linstor-drbd
volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver
linstor_uris = linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test
----

===== OpenStack Ansible

https://docs.openstack.org/openstack-ansible/[OpenStack Ansible] provides Ansible playbooks to configure and deploy of
OpenStack environments. It allows for fine-grained customization of the deployment, letting you set up the Linstor
driver directly.

./etc/openstack_ansile/user_variables.yml
----
cinder_git_repo: https://github.com/LINBIT/openstack-cinder.git
cinder_git_install_branch: linstor/stable/victoria

cinder_user_pip_packages:
  - python-linstor

cinder_backends: <1>
  linstor-drbd:
   volume_backend_name: linstor-drbd
   volume_driver: cinder.volume.drivers.linstordrv.LinstorDrbdDriver
   linstor_uris: linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test
----

<1> A detailed description of the available backend parameters can be found in the section below

===== Generic Cinder deployment

For other forms of OpenStack deployments, this guide can only provide non-specific hints.

To update the Linstor driver version, find your cinder installation. Some likely paths are:

----
/usr/lib/python3.6/dist-packages/cinder/
/usr/lib/python3.6/site-packages/cinder/
/usr/lib/python2.7/dist-packages/cinder/
/usr/lib/python2.7/site-packages/cinder/
----

The Linstor driver consists of a single file called `linstordrv.py`, located in the Cinder directory:

----
$CINDER_PATH/volume/drivers/linstordrv.py
----

To update the driver, replace the file with one from the LINBIT repository

----
RELEASE=linstor/stable/victoria
curl -fL "https://raw.githubusercontent.com/LINBIT/openstack-cinder/$RELEASE/cinder/volume/drivers/linstordrv.py" > $CINDER_PATH/volume/drivers/linstordrv.py
----

You might also need to remove the Python cache for the update to be registered:

----
rm -rf $CINDER_PATH/volume/drivers/__pycache__
----

=== Configure a Linstor Backend for Cinder

To use the Linstor driver, configure the Cinder volume service. This is done by editing the Cinder configuration file
and then restarting the Cinder Volume service.

Most of the time, the Cinder configuration file is located at `/etc/cinder/cinder.conf`. Some deployment options allow
manipulating this file in advance. See the section above for specifics.

To configure a new volume backend using Linstor, add the following section to `cinder.conf`

----
[linstor-drbd]
volume_backend_name = linstor-drbd <1>
volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver <2>
linstor_uris = linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test <3>
linstor_trusted_ca = /path/to/trusted/ca.cert <4>
linstor_client_key = /path/to/client.key <5>
linstor_client_cert = /path/to/client.cert <5>
# Deprecated or removed in 2.0.0
linstor_default_storage_pool_name = cinderpool <6>
linstor_autoplace_count = 2 <7>
linstor_controller_diskless = true <8>
# non-linstor-specific options
... <9>
----

NOTE: The parameters described here are based on the latest release provided by Linbit. The driver included in OpenStack
might not support all of these parameters. Take a look at the
https://docs.openstack.org/cinder/latest/configuration/block-storage/drivers/linstor-driver.html[OpenStack driver documentation]
to find out more.

<1> The name of the volume backend. Needs to be unique in the Cinder configuration. The whole section should share the
    same name. This name is referenced again in `cinder.conf` in the `enabled_backends` setting and when creating a new
    volume type.

<2> The version of the Linstor driver to use. There are two options:
* `cinder.volume.drivers.linstordrv.LinstorDrbdDriver`
* `cinder.volume.drivers.linstordrv.LinstorIscsiDriver`
+
Which driver you should use depends on your Linstor set up and requirements. Details on each choice are documented in
<<s-openstack-linstor-transport-options, the section below.>>

<3> The URL(s) of the Linstor Controller(s). Multiple Controllers can be specified to make use of
    <<s-linstor_ha,Linstor High Availability>>. If not set, defaults to `linstor://localhost`.
+
NOTE: In driver versions prior to 2.0.0, this option is called `linstor_default_uri`

<4> If <<s-linstor-rest-api-https,HTTPS is enabled>> the referenced certificate is used to verify the Linstor Controller
    authenticity.

<5> If <<s-linstor-rest-api-https,HTTPS is enabled>> the referenced key and certificate will be presented to the Linstor
    Controller for authentication.

<6> *Deprecated in 2.0.0, use <<s-openstack-volume-types,volume types>> instead.* The storage pools to use when placing
    resources. Applies to all diskfull resources created. Defaults to `DfltStorPool`.

<7> *Removed in 2.0.0, use <<s-openstack-volume-types,volume types>> instead.* The number of replicas to create for the given
    volume. A value of `0` will create a replica on all nodes. Defaults to `0`.

<8> *Removed in 2.0.0, volumes are created on demand by the driver.* If set to true, ensures that at least one (diskless) replica is deployed on the Cinder Controller host. This is
    useful for ISCSI transports. Defaults to `true`.

<9> You can specify more generic Cinder options here, for example `target_helper = tgtadm` for the ISCSI connector.

NOTE: You can also configure multiple Linstor backends, choosing a different name and configuration options for each.

After configuring the Linstor backend, it should also be enabled. Add it to the list of enabled backends in `cinder.conf`,
and optionally set is as the default backend:
----
[DEFAULT]
...
default_volume_type = linstor-drbd-volume
enabled_backends = lvm,linstor-drbd
...
----

As a last step, if you changed the Cinder configuration or updated the driver itself, make sure to restart the Cinder
service(s). Please check the documentation for your OpenStack Distribution on how to restart services.

[[s-openstack-linstor-transport-options]]
==== Choice of Transport Protocol

The Transport Protocol in Cinder is how clients (for example nova-compute) access the actual volumes. With Linstor, you
can choose between two different drivers that use different transports.

* `cinder.volume.drivers.linstordrv.LinstorDrbdDriver`, which uses DRBD as transport
* `cinder.volume.drivers.linstordrv.LinstorIscsiDriver`, which uses ISCSI as transport

===== Using DRBD as Transport Protocol

The `LinstorDrbdDriver` works by ensuring a replica of the volume is available locally on the node where
a client (i.e. nova-compute) issued a request. This only works if _all_ compute nodes are also running
Linstor Satellites that are part of the same Linstor cluster.

The advantages of this option are:

* Once set up, the Cinder host is no longer involved in the data path. All read and write to the volume are
  handled by the local DRBD module, which will handle replication across its configured peers.
* Since the Cinder host is not involved in the data path, any disruptions to the Cinder service do not affect
  volumes that are already attached.

====== Known limitations:

* Not all hosts and hypervisors support using DRBD volumes. This restricts deployment to Linux hosts and `kvm` hypervisors.
* Resizing of attached and in-use volumes does not fully work. While the resize itself is successful, the compute service
  will not propagate it to the VM until after a restart.
* Multi-attach (attaching the same volume on multiple VMs) is not supported.
* https://docs.openstack.org/cinder/latest/configuration/block-storage/volume-encryption.html#create-an-encrypted-volume-type[Encrypted volumes]
  only work if udev rules for DRBD devices are in place.
+
NOTE: `udev` rules are either part of the `drbd-utils` package or have their own `drbd-udev` package.

===== Using ISCSI as Transport Protocol

The default way to export Cinder volumes is via iSCSI. This brings the
advantage of maximum compatibility - iSCSI can be used with every hypervisor,
be it VMWare, Xen, HyperV, or KVM.

The drawback is that all data has to be sent to a Cinder node, to be processed
by an (userspace) iSCSI daemon; that means that the data needs to pass the
kernel/userspace border, and these transitions will cost some performance.

Another drawback is the introduction of a single point of failure. If a Cinder
node running the iSCSI daemon crashes, other nodes lose access to their volumes.
There are ways to configure Cinder for automatic fail-over to mitigate this, but
it requires considerable effort.

NOTE: In driver versions prior to 2.0.0, the Cinder host needs access to a local replica of every volume. This can be
achieved by either setting `linstor_controller_diskless=True` or using `linstor_autoplace_count=0`. Newer driver
versions will create such a volume on demand.

[[s-openstack-linstor-backend-status]]
==== Verify status of Linstor backends

To verify that all backends are up and running, you can use the OpenStack command line client:

[source,shell]
----
$ openstack volume service list
+------------------+----------------------------------------+------+---------+-------+----------------------------+
| Binary           | Host                                   | Zone | Status  | State | Updated At                 |
+------------------+----------------------------------------+------+---------+-------+----------------------------+
| cinder-scheduler | cinder-01.openstack.test               | nova | enabled | up    | 2021-03-10T12:24:37.000000 |
| cinder-volume    | cinder-01.openstack.test@linstor-drbd  | nova | enabled | up    | 2021-03-10T12:24:34.000000 |
| cinder-volume    | cinder-01.openstack.test@linstor-iscsi | nova | enabled | up    | 2021-03-10T12:24:35.000000 |
+------------------+----------------------------------------+------+---------+-------+----------------------------+
----

If you have the Horizon GUI deployed, check `Admin > System Information > Block Storage Service` instead.

In the above example all configured services are `enabled` and `up`. If there are any issues, please check
the logs of the Cinder Volume service.

[[s-openstack-volume-types]]
=== Create a new volume type for Linstor

Before creating volumes using Cinder, you have to create a volume type. This can either be done via the command line:

[source,shell]
----
# Create a volume using the default backend
$ openstack volume type create default
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| description | None                                 |
| id          | 58365ffb-959a-4d91-8821-5d72e5c39c26 |
| is_public   | True                                 |
| name        | default                              |
+-------------+--------------------------------------+
# Create a volume using a specific backend
$ openstack volume type create --property volume_backend_name=linstor-drbd linstor-drbd-volume
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| description | None                                 |
| id          | 08562ea8-e90b-4f95-87c8-821ac64630a5 |
| is_public   | True                                 |
| name        | linstor-drbd-volume                  |
| properties  | volume_backend_name='linstor-drbd'   |
+-------------+--------------------------------------+
----

Alternatively, you can create volume types via the Horizon GUI. Navigate to `Admin > Volume > Volume Types` and click
"Create Volume Type". You can assign it a backend by adding the `volume_backend_name` as "Extra Specs" to it.

==== Advanced Configuration of volume types

Each volume type can be customized by adding properties or "Extra Specs" as they are called in the Horizon GUI.

To add a property to a volume type on the command line use:
----
openstack volume type set linstor_drbd_b --property linstor:redundancy=5
----

Alternatively, you can set the property via the GUI by navigating tp `Admin > Volume > Volume Types`. In the `Actions`
column, open the dropdown menu and click the `View Extra Specs` button. This opens a dialog you can use to create, edit
and delete properties.

===== Available volume type properties

linstor:diskless_on_remaining::
Create diskless replicas on non-selected nodes after auto-placing.

linstor:do_not_place_with_regex::
Do not place the resource on a node which has a resource with a name matching the regex.

linstor:layer_list::
Comma-separated list of layers to apply for resources. If empty, defaults to DRBD,Storage.

linstor:provider_list::
Comma-separated list of providers to use. If empty, Linstor will automatically choose a suitable provider.

linstor:redundancy::
Number of replicas to create. Defaults to 2.

linstor:replicas_on_different::
A comma-separated list of key or key=value items used as autoplacement selection labels when autoplace is used to
determine where to provision storage.

linstor:replicas_on_same::
A comma-separated list of key or key=value items used as autoplacement selection labels when autoplace is used to
determine where to provision storage.

linstor:storage_pool::
Comma-separated list of storage pools to use when auto-placing.

linstor:property:*::
If a key is prefixed by `linstor:property:`, it is interpreted as a LINSTOR property. The property gets set on the <<s-linstor-resource-groups,Resource Group>>
created for the volume type.
+
For example: To change the <<s-linstor-auto-quorum,quorum policy>> `DrbdOptions/auto-quorum` needs to be set. This can
be done by setting the `linstor:property:DrbdOptions/auto-quorum` property.

=== Using volumes

Once you have a volume type configured, you can start using it to provision new volumes.

For example, to create a simple 1Gb volume on the command line you can use:
[source,shell]
----
openstack volume create --type linstor-drbd-volume --size 1 --availability-zone nova linstor-test-vol
openstack volume list
----

NOTE: If you set `default_volume_type = linstor-drbd-volume` in your `/etc/cinder/cinder.conf`,
you may omit the `--type linstor-drbd-volume` from the `openstack volume create ...` command above.

=== Troubleshooting

This section describes what to do in case you encounter problems with using Linstor volumes and snapshots.

==== Checking for error messages in Horizon

Every volume and snapshot has a Messages tab in the Horizon dashboard. In case of errors, the list of messages can
be used as a starting point for further investigation. Some common messages in case of errors:

  create volume from backend storage:Driver failed to create the volume.

There was an error creating a new volume. Check the Cinder Volume service logs for more details

  schedule allocate volume:Could not find any available weighted backend.

If this is the only error message, this means the Cinder Scheduler could not find a volume backend suitable for creating
the volume. This is most likely because:

* The volume backend is offline. See <<s-openstack-linstor-backend-status>>
* The volume backend has not enough free capacity to fulfil the request. Check the output of `cinder get-pools --detail`
  and `linstor storage-pool list` to ensure that the requested capacity is available.

==== Checking the Cinder Volume Service

The Linstor driver is called as part of the Cinder Volume service.

[frame="topbot",options="header"]
|====
| Distribution | Log location or command
| DevStack     | `journalctl -u devstack@c-vol`
|====

==== Checking the compute srvice logs

Some issues will not be logged in the Cinder Service but in the actual consumer of the volumes, most likely the compute
service (Nova). As with the volume service, the exact host and location to check depends on your Openstack distribution:

[frame="topbot",options="header"]
|====
| Distribution | Log location or command
| DevStack     | `journalctl -u devstack@n-cpu`
|====

// Keep the empty line before this comment, otherwise the next chapter is folded into this
