[[ch-proxmox-linstor]]
== LINSTOR Volumes in Proxmox VE

indexterm:[Proxmox]This chapter describes DRBD in Proxmox VE via
the http://git.linbit.com/linstor-proxmox.git[LINSTOR Proxmox Plugin].

[[s-proxmox-ls-overview]]
=== Proxmox VE Overview

http://www.proxmox.com/en/[Proxmox VE] is an easy to use, complete server
virtualization environment with KVM, Linux Containers and HA.

'linstor-proxmox' is a Perl plugin for Proxmox that, in combination with LINSTOR, allows to replicate VM
//(LVM volumes on DRBD)
disks  on several Proxmox VE nodes. This allows to live-migrate
active VMs within a few seconds and with no downtime without needing a central SAN, as the data is already
replicated to multiple nodes.

[[s-proxmox-ls-upgrades]]
=== Upgrades
If this is a fresh installation, skip this section and continue with <<s-proxmox-ls-install>>.

[[s-proxmox-ls-upgrades-4-5]]
==== From 4.x to 5.x
Version 5 of the plugin drops compatibility with the legacy configuration options "storagepool" and
"redundancy". Version 5 *requires* a "resourcegroup" option, and obviously a LINSTOR resource group. The old
options should be removed from the config.

Configuring LINSTOR is described in Section <<s-proxmox-ls-ls-configuration>>, a typical example follows:
Let's assume the pool was set to "mypool", and redundancy to 3.
----------------------------
# linstor resource-group create --storage-pool=mypool --place-count=3 drbdMypoolThree
# linstor volume-group create drbdMypoolThree
# vi /etc/pve/storage.cfg
drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13
   resourcegroup drbdMypoolThree
----------------------------

[[s-proxmox-ls-upgrades-5-6]]
==== From 5.x to 6.x
With version 6 PVE added additional parameters to some functions and rightfully reset their "APIAGE". This
means that old plugins, while actually usable as they don't use any of these changed functions do not work
anymore. Please upgrade to plugin version 5.2.1 at least.

[[s-proxmox-ls-install]]
=== Proxmox Plugin Installation

LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the
Proxmox plugin, but the whole DRBD SDS stack including a DRBD SDS kernel
module and user space utilities.

The DRBD9 kernel module is installed as a `dkms` package (i.e., `drbd-dkms`), therefore you'll have to install
`pve-headers` package, before you set up/install the software packages from LINBIT's repositories. Following
that order, ensures that the kernel module will build properly for your kernel. If you don't plan to install
the latest Proxmox kernel, you have to install kernel headers matching your current running kernel (e.g.,
		`pve-headers-$(uname -r)`). If you missed this step, then still you can rebuild the dkms package against
your current kernel, (kernel headers have to be installed in advance), by issuing `apt install --reinstall
drbd-dkms` command.

LINBIT's repository can be enabled as follows, where "$PVERS" should be set to your Proxmox VE *major version*
(e.g., "7", not "7.1"):

----------------------------
# wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add -
# PVERS=7 && echo "deb http://packages.linbit.com/proxmox/ proxmox-$PVERS drbd-9" > \
	/etc/apt/sources.list.d/linbit.list
# apt update && apt install linstor-proxmox
----------------------------

[[s-proxmox-ls-ls-configuration]]
=== LINSTOR Configuration
For the  rest of this guide we assume that you have a LINSTOR cluster configured as described in
<<s-linstor-init-cluster>>. Start the "linstor-controller" on one node, and the "linstor-satellite" on all
nodes. The preferred way to use the plugin, starting from version 4.1.0, is via LINSTOR resource groups and a
single volume group within every resource group. LINSTOR resource groups are described in
<<s-linstor-resource-groups>>. All the required LINSTOR configuration (e.g., redundancy count) has to be set
on the resource group.

[[s-proxmox-ls-configuration]]
=== Proxmox Plugin Configuration
The final step is to provide a configuration for Proxmox itself. This can be done by adding an entry in the
`/etc/pve/storage.cfg` file, with a content similar to the following.

----------------------------
drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13
   resourcegroup defaultpool
----------------------------

The "drbd" entry is fixed and you are not allowed to modify it, as it tells to Proxmox to use DRBD as storage
backend.  The "drbdstorage" entry can be modified and is used as a friendly name that will be shown in the PVE
web GUI to locate the DRBD storage.  The "content" entry is also fixed, so do not change it. The redundancy
(specified in the resource group) specifies how many replicas of the data will be stored in the cluster. The recommendation is to set it
to 2 or 3 depending on your setup.  The data is accessible from all nodes, even
if some of them do not have local copies of the data.  For example, in a 5 node cluster, all nodes will be
able to access 3 copies of the data, no matter where they are stored in. The "controller" parameter must be
set to the IP of the node that runs the LINSTOR controller service. Only one node can be set to run as LINSTOR
controller at the same time.  If that node fails, start the LINSTOR controller on another node and change that
value to its IP address.

A configuration using different storage pools in different resource groups would look like this:

----------------------------
drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13
   resourcegroup defaultpool

drbd: fastdrbd
   content images,rootdir
   controller 10.11.12.13
   resourcegroup ssd

drbd: slowdrbd
   content images,rootdir
   controller 10.11.12.13
   resourcegroup backup
----------------------------

By now, you should be able to create VMs via Proxmox's web GUI by selecting "__drbdstorage__", or any other of
the defined pools as storage location.

Starting from version 5 of the plugin one can set the option "preferlocal yes". If it is set, the plugin tries
to create a diskful assignment on the node that issued the storage create command. With this option one can
make sure the VM gets local storage if possible. Without that option LINSTOR might place the storage on nodes
'B' and 'C', while the VM is initially started on node 'A'. This would still work as node 'A' then would get a
diskless assignment, but having local storage might be preferred.

.NOTE: DRBD supports only the **raw** disk format at the moment.

At this point you can try to live migrate the VM - as all data is accessible on all nodes (even on Diskless
nodes) - it will take just a few seconds. The overall process might take a bit longer if the VM is under
load and if there is a lot of RAM being dirtied all the time.  But in any case, the downtime should be minimal
and you will see no interruption at all.

.Table Configuration Options
|===
| Option | Meaning

| `controller`    | The IP of the LINSTOR controller (',' separated list allowed)
| `resourcegroup` | The name of a LINSTOR resource group which defines the deployment of new VMs. As described above
| `preferlocal`   | Prefer to create local storage (yes/no). As decribed above 
| `statuscache`   | Time in seconds status information is cached, 0 means no extra cache. Relevant on huge clusters with hundreds of resources. This has to be set on *all* `drbd` storages in `/etc/pve/storage.cfg` to take effect.
| `apicrt`        | Path to the client certificate
| `apikey`        | Path to the client private key
| `apica`         | Path to the CA certificate
|===

[[s-proxmox-ls-HA]]
=== Making the Controller Highly-Available (optional)
Making LINSTOR highly-available is a matter of making the LINSTOR controller
highly-available. This step is described in Section <<s-linstor_ha>>.

The last -- but crucial -- step is to configure the Proxmox plugin to be
able to connect to multiple LINSTOR controllers. It will use the first one it
receives an answer from. This is done by adding a comma-separated list of
controllers in the `controller` section of the plugin like this:

----------------------------
drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13,10.11.12.14,10.11.12.15
   resourcegroup defaultpool
----------------------------

