[[s-administrative-tasks-setup]]
== Basic administrative tasks / Setup

LINSTOR is a configuration management system for storage on Linux systems.
It manages LVM logical volumes and/or ZFS ZVOLs on a cluster of nodes. It
leverages DRBD for replication between different nodes and to provide
block storage devices to users and applications. It manages snapshots,
encryption and caching of HDD backed data in SSDs via bcache.

// Troubleshooting for LINSTOR guide?
/////
This chapter outlines typical administrative tasks encountered during
day-to-day operations. It does not cover troubleshooting tasks, these
are covered in detail in <<ch-troubleshooting>>.
/////


[[s-concepts_and_terms]]
=== Concepts and Terms
This section goes over core concepts and terms that you will need to familiarize yourself
with to understand how LINSTOR works and deploys storage. The section is laid out in a
"ground up" approach.


==== Installable Components
===== linstor-controller
A LINSTOR setup requires at least one active controller and one or more satellites.

// Once the chapter on making your controller HA is done we need to link that here

The _linstor-controller_ relies on a database that holds all configuration
information for the whole cluster. It makes all decisions that need to have a
view of the whole cluster. Multiple controllers can be used for LINSTOR but
only one can be active.

===== linstor-satellite
The _linstor-satellite_ runs on each node where LINSTOR consumes local
storage or provides storage to services. It is stateless; it receives
all the information it needs from the controller. It runs programs
like `lvcreate` and `drbdadm`. It acts like a node agent.

===== linstor-client
The _linstor-client_ is a command line utility that you use to issue
commands to the system and to investigate the status of the system.

==== Objects
Objects are the end result which LINSTOR presents to the end-user or application,
such as; Kubernetes/OpenShift, a replicated block device (DRBD), NVMeOF target, etc.

===== Node
Node's are a server or container that participate in a LINSTOR cluster. The _Node_
attribute defines:

* Determines which LINSTOR cluster the node participates in
* Sets the role of the node: Controller, Satellite, Auxiliary
* *NetInterface* objects define the node's connectivity

===== NetInterface
As the name implies, this is how you define the interface/address of a node's network interface.

===== Definitions
Definitions define attributes of an object, they can be thought of as
profile or template. Objects created will inherit the configuration
defined in the definitions. A definition must be defined prior to creating
the associated object.  For example; you must create a _ResourceDefinition_
prior to creating the _Resource_

StoragePoolDefinition :::
* Defines the name of a storage pool

ResourceDefinition :::
Resource definitions define the following attributes of a resource:
* The name of a DRBD resource
* The TCP port for DRBD to use for the resource's connection

VolumeDefinition :::
Volume definitions define the following:

* A volume of a DRBD resource
* The size of the volume
* The volume number of the DRBD resource's volume
* The meta data properties of the volume
* The minor number to use for the DRBD device associated with the DRBD volume

===== StoragePool
The _StoragePool_ identifies storage in the context of LINSTOR. It defines:

* The configuration of a storage pool on a specific node
* The storage back-end driver to use for the storage pool on the cluster node (LVM, ZFS, etc)
* The parameters and configuration to pass to the storage backed driver

===== Resource
LINSTOR has now has expanded its capabilities to manage a broader set of storage technologies
outside of just DRBD. A _Resource_:

* Represents the placement of a DRBD resource, as defined within the _ResourceDefinition_
* Places a resource on a node in the cluster
* Defines the placement of a _ResourceDefinition_ on a node

===== Volume
Volumes are a subset of a _Resource_. A _Resource_ could have multiple volumes, for example
you may wish to have your database stored on slower storage than your logs in your MySQL cluster.
By keeping the _volumes_ under a single _resource_ you are essentially creating a consistency group.
The _Volume_ attribute can define also define attributes on a more granular level.

[[s-broader_context]]
=== Broader Context

While LINSTOR might be used to make the management of DRBD more
convenient, it is often integrated with software stacks higher up.
Such integration exist already for Kubernetes, OpenStack, OpenNebula
and Proxmox. Chapters specific to deploying LINSTOR in these
environments are included in this guide.

The southbound drivers used by LINSTOR are LVM, thinLVM and ZFS.

[[s-packages]]
=== Packages

LINSTOR is packaged in both the .rpm and the .deb variants:

. _linstor-client_ contains the command line client program. It depends
  on python which is usually already installed. In RHEL 8 systems you will need to symlink
python
. _linstor-controller_  and _linstor-satellite_ Both contain systemd unit files
for the services. They depend on Java runtime environment (JRE) version 1.8
(headless) or higher.

For further detail about these packages see the
<<Installable Components,Installable Components>> section above.

NOTE: If you have a support subscription to LINBIT, you will have access to
our certified binaries via our repositories.

[[s-installation]]
=== Installation

IMPORTANT: If you want to use LINSTOR in containers skip this Topic and use the "Containers" section below for the installation.



[[s-admin-linstor-ubuntu_linux]]
==== Ubuntu Linux

If you want to have the option of creating replicated storage using DRBD, you will need to install _drbd-dkms_ and _drbd-utils_. These packages will need to be installed on all nodes. You will also need
to choose a volume manager, either ZFS or LVM, in this instance we're using LVM.

--------------------------------------------
# apt install -y drbd-dkms drbd-utils lvm2
--------------------------------------------

Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package.

Combined node:

--------------------------------------------------------------------
# apt install linstor-controller linstor-satellite  linstor-client
--------------------------------------------------------------------

That will make our remaining nodes our Satellites, so we'll need to install the following packages on them:

-------------------------------------------------
# apt install linstor-satellite  linstor-client
-------------------------------------------------


[[s-admin-suse_linux_enterprise_server]]
==== SUSE Linux Enterprise Server


SLES High Availability Extension (HAE) includes DRBD.

On SLES, DRBD is normally installed via the software installation component of YaST2. It comes bundled with the High Availability
package selection.

As we download DRBD's newest module we can check if the LVM-tools are up to date as well. User who prefer a command line install
may simply issue the following command to get the newest DRBD and LVM version:

----------------------
# zypper install drbd lvm2
----------------------



Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages
are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package.

Combined node:

------------------------------------------------------------------------
# zypper install linstor-controller linstor-satellite  linstor-client
------------------------------------------------------------------------

That will make our remaining nodes our Satellites, so we'll need to install the following packages on them:

----------------------------------------------------
# zypper install linstor-satellite  linstor-client
----------------------------------------------------


[[s-admin-centos]]
==== CentOS

CentOS has had DRBD 8 since release 5. For DRBD 9 you'll need to look at EPEL and similar sources.
Alternatively, if you have an active support contract with LINBIT you can utilize our RHEL 8 repositories.
DRBD can be installed using `yum`. We can also check for the newest version of the LVM-tools as well.

IMPORTANT: LINSTOR *requires* DRBD 9 if you wish to have replicated storage. This requires an external
repository to be configured, either LINBIT's or a 3rd parties.

-----------------------------
# yum install drbd kmod-drbd lvm2
-----------------------------

Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages
are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package.

NOTE: On RHEL 8 systems you will need to install python2 for the linstor-client to work.

Combined node:

-------------------------------------------------------------------
# yum install linstor-controller linstor-satellite  linstor-client
-------------------------------------------------------------------

That will make our remaining nodes our Satellites, so we'll need to install the following packages on them:

------------------------------------------------
# yum install linstor-satellite  linstor-client
------------------------------------------------


[[s-upgrading]]
=== Upgrading

LINSTOR doesn't support rolling upgrade, controller and satellites must have the same version, otherwise
the controller with discard the satellite with a `VERSION_MISMATCH`.
But this isn't a problem, as the satellite won't do any actions as long it isn't connected to a controller
and DRBD will not be disrupted by any means.

If you are using the embedded default H2 database and the linstor-controller package is upgraded an automatic
backup file of the database will be created in the default `/var/lib/linstor` directory.
This file is a good restore point if for any reason a linstor-controller database migration should fail,
than it is recommended to report the error to LINBIT and restore the old database file and downgrade to your previous
controller version.

If you use any external database or etcd, it is recommended to do a manually backup of your current database to have
a restore point.

So first upgrade the `linstor-controller`, `linstor-client` package on you controller host and restart the `linstor-controller`,
the controller should start and all of it's client should show `OFFLINE(VERSION_MISMATCH)`.
After that you can continue upgrading `linstor-satellite` on all satellite nodes and restart them, after a short reconnection time
they should all show `ONLINE` again and your upgrade is finished.


[[s-containers]]
=== Containers

LINSTOR is also available as containers. The base images are available
in LINBIT's container registry, `drbd.io`.

In order to access the images, you first have to login to the
registry (reach out to sales@linbit.com for credentials):

----------------------------
# docker login drbd.io
----------------------------

The containers available in this repository are:

* drbd.io/drbd9-rhel8
* drbd.io/drbd9-rhel7
* drbd.io/drbd9-sles15sp1
* drbd.io/drbd9-bionic
* drbd.io/drbd9-focal
* drbd.io/linstor-csi
* drbd.io/linstor-controller
* drbd.io/linstor-satellite
* drbd.io/linstor-client

An up to date list of available images with versions can be retrieved by opening http://drbd.io in your
browser. Make sure to access the host via "http", as the registry's images themselves are served via "https".

To load the kernel module, needed only for LINSTOR satellites, you'll need to
run a `drbd9-$dist` container in privileged mode. The kernel module containers
either retrieve an official LINBIT package from a customer repository, use
shipped packages, or they try to build the kernel modules from source. If you
intend to build from source, you need to have the according kernel headers
(e.g., `kernel-devel`) installed on the host. There are 4 ways to execute such
a module load container:

* Building from shipped source
* Using a shipped/pre-built kernel module
* Specifying a LINBIT node hash and a distribution.
* Bind-mounting an existing repository configuration.

Example building from shipped source (RHEL based):

----------------------------
# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  -v /usr/src:/usr/src:ro \
  drbd.io/drbd9-rhel7
----------------------------

Example using a module shipped with the container, which is enabled by *not* bind-mounting `/usr/src`:

----------------------------
# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  drbd.io/drbd9-rhel8
----------------------------

Example using a hash and a distribution (rarely used):

----------------------------
# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  -e LB_DIST=rhel7.7 -e LB_HASH=ThisIsMyNodeHash \
  drbd.io/drbd9-rhel7
----------------------------

Example using an existing repo config (rarely used):

----------------------------
# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  -v /etc/yum.repos.d/linbit.repo:/etc/yum.repos.d/linbit.repo:ro \
  drbd.io/drbd9-rhel7
----------------------------

IMPORTANT: In both cases (hash + distribution, as well as bind-mounting a repo)
the hash or config has to be from a node that has a special property set. Feel
free to contact our support, and we set this property.

IMPORTANT: For now (i.e., pre DRBD 9 version "9.0.17"), you must use the containerized DRBD kernel module,
as opposed to loading a kernel module onto the host system. If you
intend to use the containers you should not install the DRBD kernel
module on your host systems. For DRBD version 9.0.17 or greater, you can install the kernel module as usual on
the host system, but you need to make sure to load the module with the `usermode_helper=disabled` parameter
(e.g., `modprobe drbd usermode_helper=disabled`).

Then run the LINSTOR satellite container, also privileged, as a daemon:

----------------------------
# docker run -d --name=linstor-satellite --net=host -v /dev:/dev --privileged drbd.io/linstor-satellite
----------------------------

NOTE: `net=host` is required for the containerized `drbd-utils` to be
able to communicate with the host-kernel via netlink.

To run the LINSTOR controller container as a daemon, mapping ports
`3370`, `3376` and `3377` on the host to the container:

----------------------------
# docker run -d --name=linstor-controller -p 3370:3370 -p 3376:3376 -p 3377:3377 drbd.io/linstor-controller
----------------------------

To interact with the containerized LINSTOR cluster, you can either use
a LINSTOR client installed on a system via packages, or via the
containerized LINSTOR client. To use the LINSTOR client container:

----------------------------
# docker run -it --rm -e LS_CONTROLLERS=<controller-host-IP-address> drbd.io/linstor-client node list
----------------------------

From this point you would use the LINSTOR client to initialize your
cluster and begin creating resources using the typical LINSTOR
patterns.

To stop and remove a daemonized container and image:

----------------------------
# docker stop linstor-controller
# docker rm linstor-controller
----------------------------

[[s-linstor-init-cluster]]
=== Initializing your cluster
We assume that the following steps are accomplished on *all* cluster nodes:

. The DRBD9 kernel module is installed and loaded
. `drbd-utils` are installed
. `LVM` tools are installed
. `linstor-controller` and/or `linstor-satellite` its dependencies are installed
. The `linstor-client` is installed on the `linstor-controller` node

Start and enable the linstor-controller service on the host where it has been installed:

----------------------------
# systemctl enable --now linstor-controller
----------------------------

If you are sure the linstor-controller service gets automatically enabled on installation you can use the
following command as well:

------------------------------------------
# systemctl start linstor-controller
------------------------------------------

[[s-using_the_linstor_client]]
=== Using the LINSTOR client
Whenever you run the LINSTOR command line client, it needs to know where your
linstor-controller runs. If you do not specify it, it will try to reach a locally
running linstor-controller listening on IP `127.0.0.1` port `3376`. Therefore we
will use the `linstor-client` on the same host as the `linstor-controller`.

IMPORTANT: The `linstor-satellite` requires ports 3366 and 3367. The `linstor-controller`
requires ports 3376 and 3377. Make sure you have these ports allowed on your firewall.

----------------------------
# linstor node list
----------------------------
should give you an empty list and not an error message.

You can use the `linstor` command on any other machine, but then you need
to tell the client how to find the linstor-controller. As shown, this can be
specified as a command line option, an environment variable, or in a global
file:

----------------------------
# linstor --controllers=alice node list
# LS_CONTROLLERS=alice linstor node list
----------------------------

Alternatively you can create the `/etc/linstor/linstor-client.conf`
file and populate it like below.


-----
[global]
controllers=alice
-----

If you have multiple linstor-controllers configured you can simply
specify them all in a comma separated list. The linstor-client will
simply try them in the order listed.


NOTE: The linstor-client commands can also be used in a much faster
and convenient way by only writing the starting letters of the parameters
e.g.: `linstor node list` -> `linstor n l`

[[s-adding_nodes_to_your_cluster]]
=== Adding nodes to your cluster
The next step is to add nodes to your LINSTOR cluster.

----------------------------
# linstor node create bravo 10.43.70.3
----------------------------

If the IP is omitted, the client will try to resolve the given node-name as
host-name by itself.

Linstor will automatically detect the node's local `uname -n` which is
later used for the DRBD-resource.

When you use `linstor node list` you will see that the new node
is marked as offline. Now start and enable the linstor-satellite on that node
so that the service comes up on reboot as well:
----------------------------
# systemctl enable --now  linstor-satellite
----------------------------

You can also use `systemctl start linstor-satellite`
if you are sure that the service is already enabled as default and comes up on
reboot.

About 10 seconds later you will see the status in `linstor node list`
becoming online. Of course the satellite process may be started before
the controller knows about the existence of the satellite node.

NOTE: In case the node which hosts your controller should also contribute
storage to the LINSTOR cluster, you have to add it as a node and start
the linstor-satellite as well.

If you want to have other services wait until the linstor-satellite had a chance
to create the necessary devices (i.e. after a boot), you can update the
corresponding .service file and change `Type=simple` to `Type=notify`.

This will cause the satellite to delay sending the `READY=1` message to systemd
until the controller connects, sends all required data to the satellite and the
satellite at least tried once to get the devices up and running.

[[s-storage_pools]]
=== Storage pools

<<StoragePool,StoragePools>> identify storage in the context of LINSTOR.
To group storage pools from multiple nodes, simply use the same name
on each node.
For example, one valid approach is to give all SSDs one name and
all HDDs another.

On each host contributing storage, you need to create
either an LVM VG or a ZFS zPool. The VGs and zPools identified with one
LINSTOR storage pool name may have different VG or zPool names on the
hosts, but do yourself a favor and use the same VG or zPool name on all
nodes.

----------------------------
# vgcreate vg_ssd /dev/nvme0n1 /dev/nvme1n1 [...]
----------------------------

These then need to be registered with LINSTOR:

----------------------------
# linstor storage-pool create lvm alpha pool_ssd vg_ssd
# linstor storage-pool create lvm bravo pool_ssd vg_ssd
----------------------------

NOTE: The storage pool name and common metadata is referred to as a
_storage pool definition_.
The listed commands create a storage pool definition implicitly.
You can see that by using `linstor storage-pool-definition list`.
Creating storage pool definitions explicitly is possible but
not necessary.

To list your storage-pools you can use:

------
# linstor storage-pool list
------

or using the short version

-----
# linstor sp l
-----

////
In case anything goes wrong with the storage pool's VG/zPool, e.g. the VG having been renamed or somehow
became invalid you can delete the storage pool in LINSTOR with the following command, given that only
resources with all their volumes in the so-called 'lost' storage pool are attached. This feature is available
since LINSTOR v0.9.13.

------
# linstor storage-pool lost alpha pool_ssd
------

or using the short version

-----
# linstor sp lo alpha pool_ssd
-----
////

Should the deletion of the storage pool be prevented due to attached resources or snapshots with some of its
volumes in another still functional storage pool, hints will be given in the 'status' column of the
corresponding list-command (e.g. `linstor resource list`). After deleting the LINSTOR-objects in the lost storage pool
manually, the lost-command can be executed again to ensure a complete deletion of the storage pool and its
remaining objects.

[[s-a_storage_pool_per_backend_device]]
==== A storage pool per backend device

In clusters where you have only one kind of storage and the capability
to hot-repair storage devices, you may choose a model where you create
one storage pool per physical backing device. The advantage of this
model is to confine failure domains to a single storage device.

[[s-physical-storage-command]]
==== Physical storage command

Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you.
The linstor-client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools
are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes.

-----
# linstor physical-storage list
-----

Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk).

It will only show disks that pass the following filters:

  * The device size must be greater than 1GiB
  * The device is a root device (not having children) e.g.: /dev/vda, /dev/sda
  * The device does not have any file-system or other `blkid` marker (`wipefs -a` might be needed)
  * The device is no DRBD device


With the `create-device-pool` command you can create a LVM pool on a disk and also directly
add it as a storage-pool in LINSTOR.

----
# linstor physical-storage create-device-pool --pool-name lv_my_pool LVMTHIN node_alpha /dev/vdc --storage-pool newpool
----

If the `--storage-pool` option was provided, LINSTOR will create a storage-pool with the given name.

For more options and exact command usage please check the linstor-client help.

[[s-linstor-resource-groups]]
=== Resource groups

A resource group is a parent object of a resource definition where all
property changes made on a resource group will be inherited by it's
resource definition children. The resource group also stores settings
for automatic placement rules and can spawn a resource definition
depending on the stored rules.

In simpler terms, resource groups are like templates that define
characteristics of resources created from them. Changes to these
pseudo templates will be applied to all resources that were created
from the resource group, retroactively.

TIP: Using resource groups to define how you'd like your resources
provisioned should be considered the de facto method for deploying
volumes provisioned by LINSTOR. Chapters that follow which describe
creating each _resource_ from a _resource-definition_ and
_volume-definition_ should only be used in special scenarios.

NOTE: Even if you choose not to create and use _resource-groups_ in
your LINSTOR cluster, all resources created from
_resource-definitions_ and _volume-definitions_ will exist in the
'DfltRscGrp' _resource-group_.

A simple pattern for deploying resources using resource groups would
look like this:

----
# linstor resource-group create my_ssd_group --storage-pool pool_ssd --place-count 2
# linstor volume-group create my_ssd_group
# linstor resource-group spawn-resources my_ssd_group my_ssd_res 20G
----

The commands above would result in a resource named 'my_ssd_res' with a
20GB volume replicated twice being automatically provisioned from nodes who
participate in the storage pool named 'pool_ssd'.

A more useful pattern could be to create a resource group with
settings you've determined are optimal for your use case. Perhaps
you have to run nightly online verifications of your volumes'
consistency, in that case, you could create a resource group with the
'verify-alg' of your choice already set so that resources spawned from
the group are pre-configured with 'verify-alg' set:

----
# linstor resource-group create my_verify_group --storage-pool pool_ssd --place-count 2
# linstor resource-group drbd-options --verify-alg crc32c my_verify_group
# linstor volume-group create my_verify_group
# for i in {00..19}; do
    linstor resource-group spawn-resources my_verify_group res$i 10G
  done
----

The commands above result in twenty 10GiB resources being created each
with the 'crc32c' 'verify-alg' pre-configured.

You can tune the settings of individual resources or volumes spawned
from resource groups by setting options on the respective
_resource-definition_ or _volume-definition_. For example, if 'res11'
from the example above is used by a very active database receiving
lots of small random writes, you might want to increase the
'al-extents' for that specific resource:

----
# linstor resource-definition drbd-options --al-extents 6007 res11
----

If you configure a setting in a _resource-definition_ that is already
configured on the _resource-group_ it was spawned from, the value set
in the _resource-definition_ will override the value set on the parent
_resource-group_. For example, if the same 'res11' was required to use
the slower but more secure 'sha256' hash algorithm in its
verifications, setting the 'verify-alg' on the _resource-definition_
for 'res11' would override the value set on the _resource-group_:

----
# linstor resource-definition drbd-options --verify-alg sha256 res11
----

TIP: A rule of thumb for the hierarchy in which settings are inherited
is the value "closer" to the resource or volume wins: _volume-definition_
settings take precedence over _volume-group_ settings, and
_resource-definition_ settings take precedence over _resource-group_
settings.


[[s-linstor-set-config]]
=== Cluster configuration

[[s-available_storage_plugins]]
==== Available storage plugins

indexterm:[linstor, storage plugins]

LINSTOR has the following supported storage plugins as of writing:

  * Thick LVM

  * Thin LVM with a single thin pool

  * Thick ZFS

  * Thin ZFS

[[s-linstor-new-volume]]
=== Creating and deploying resources/volumes
In the following scenario we assume that the goal is to create a resource
'backups' with a size of '500 GB' that is replicated among three cluster nodes.

First, we create a new resource definition:

----------------------------
# linstor resource-definition create backups
----------------------------

Second, we create a new volume definition within that resource definition:

----------------------------
# linstor volume-definition create backups 500G
----------------------------

If you want to change the size of the volume-definition you can simply do that by:

-------
# linstor volume-definition set-size backups 0 100G
-------

The parameter `0` is the number of the volume in the resource `backups`. You have to provide this parameter
, because resources can have multiple volumes and they are identified by a so called volume-number. This number
can be found by listing the volume-definitions.

IMPORTANT: The size of a volume-definition can only be decreased if it has no resource. Despite
of that the size can be increased even with an deployed resource.

So far we have only created objects in LINSTOR's database, not a single LV was
created on the storage nodes. Now you have the choice of delegating the
task of placement to LINSTOR or doing it yourself.

[[s-manual_placement]]
==== Manual placement

With the `resource create` command you may assign a resource definition
to named nodes explicitly.

----------------------------
# linstor resource create alpha backups --storage-pool pool_hdd
# linstor resource create bravo backups --storage-pool pool_hdd
# linstor resource create charlie backups --storage-pool pool_hdd
----------------------------

[[s-autoplace-linstor]]
==== Autoplace

The value after autoplace tells LINSTOR how many replicas you want to have.
The storage-pool option should be obvious.
----------------------------
# linstor resource create backups --auto-place 3 --storage-pool pool_hdd
----------------------------
Maybe not so obvious is that you may omit the `--storage-pool` option, then
LINSTOR may select a storage pool on its own. The selection follows these rules:

  * Ignore all nodes and storage pools the current user has no access to
  * Ignore all diskless storage pools
  * Ignore all storage pools not having enough free space

The remaining storage pools will be rated by different strategies.
LINSTOR has currently three strategies:

  * `MaxFreeSpace`: This strategy maps the rating 1:1 to the remaining free
space of the storage pool. However, this strategy only considers the actually
allocated space (in case of thinly provisioned storage pool this might grow
with time without creating new resources)
  * `MinReservedSpace`: Unlike the "MaxFreeSpace", this strategy considers the
reserved spaced. That is the space that a thin volume can grow to before reaching
its limit. The sum of reserved spaces might exceed the storage pools capacity,
which is as overprovisioning.
  * `MinRscCount`: Simply the count of resources already deployed in a given
storage pool
  * `MaxThroughput`: For this strategy, the storage pool's
`Autoplacer/MaxThroughput` property is the base of the score, or 0 if the property
is not present. Every Volume deployed in the given storage pool will subtract
its defined `sys/fs/blkio_throttle_read` and `sys/fs/blkio_throttle_write` property-
value from the storage pool's max throughput. The resulting score might be negative.

The scores of the strategies will be normalized, weighted and summed up, where
the scores of minimizing strategies will be converted first to allow an overall
maximization of the resulting score.

The weights of the strategies can be configured with

----
linstor controller set-property Autoplacer/Weights/$name_of_the_strategy $weight
----
whereas the strategy-names are listed above and the weight can be an arbitrary
decimal.

NOTE: To keep the behaviour of the autoplacer similar to the old one (due to
compatibility), all strategies have a default-weight of 0, except the `MaxFreeSpace`
which has a weight of 1.

NOTE: Neither 0 nor a negative score will prevent a storage pool from getting
selected, just making them to be considered later.

Finally LINSTOR tries to find the best matching group of storage pools meeting
all requirements. This step also considers other autoplacement restrictions
as `--replicas-on-same`, `--replicas-on-different` and others.

These two arguments, `--replicas-on-same` and `--replicas-on-different` expect the
name of a property within the `Aux/` namespace. The following example shows that the
client automatically prefixes the `testProperty` with the `Aux/` namespace.

----
linstor resource-group create testRscGrp --replicas-on-same testProperty
SUCCESS:
Description:
    New resource group 'testRscGrp' created.
Details:
    Resource group 'testRscGrp' UUID is: 35e043cb-65ab-49ac-9920-ccbf48f7e27d

linstor resource-group list
+-----------------------------------------------------------------------------+
| ResourceGroup | SelectFilter                         | VlmNrs | Description |
|-============================================================================|
| DfltRscGrp    | PlaceCount: 2                        |        |             |
|-----------------------------------------------------------------------------|
| testRscGrp    | PlaceCount: 2                        |        |             |
|               | ReplicasOnSame: ['Aux/testProperty'] |        |             |
+-----------------------------------------------------------------------------+
----

NOTE: If everything went right the DRBD-resource has now been created by LINSTOR.
This can be checked by looking for the DRBD block device with the `lsblk`
command which should look like `drbd0000` or similar.


Now we should be able to mount the block device of our resource and start using
LINSTOR.

[[s-more-about-linstor]]
== Further LINSTOR tasks

[[s-linstor_ha]]
=== LINSTOR high availability
By default a LINSTOR cluster consists of exactly one LINSTOR controller. Making LINSTOR highly-available
involves providing replicated storage for the controller database, multiple LINSTOR controllers where only one
is active at a time, and a service manager that takes care of mounting/unmounting the highly-available storage
and starting/stopping LINSTOR controllers.

==== Highly-available Storage
For configuring the highly-available storage we use LINSTOR itself. This has the advantage that the storage is
under LINSTOR control and can for example be easily extended to new cluster nodes. Just create a new
resource with 200MB in size. This could look like this, you certainly need to adapt the storage pool name:

----------------------------
# linstor resource-definition create linstor_db
# linstor resource-definition drbd-options --on-no-quorum=io-error linstor_db
# linstor resource-definition drbd-options --auto-promote=no linstor_db
# linstor volume-definition create linstor_db 200M
# linstor resource create linstor_db -s pool1 --auto-place 3
----------------------------

From now on we assume the resource's name is "linstor_db". It is crucial that your
cluster qualifies for auto-quorum and uses the `io-error` policy (see Section <<s-linstor-auto-quorum>>), and
that `auto-promote` is disabled.

After the resource is created, it is time to move the LINSTOR DB to the new storage and to create a `systemd`
mount service. First we stop the current controller and disable it, as it will be managed by `drbd-reactor` later.

----------------------------
# systemctl disable --now linstor-controller

# cat << EOF > /etc/systemd/system/var-lib-linstor.mount
[Unit]
Description=Filesystem for the LINSTOR controller

[Mount]
# you can use the minor like /dev/drbdX or the udev symlink
What=/dev/drbd/by-res/linstor_db/0
Where=/var/lib/linstor
EOF

# mv /var/lib/linstor{,.orig}
# mkdir /var/lib/linstor
# chattr +i /var/lib/linstor  # only if on LINSTOR >= 1.14.0
# drbdadm primary linstor_db
# mkfs.ext4 /dev/drbd/by-res/linstor_db/0
# systemctl start var-lib-linstor.mount
# cp -r /var/lib/linstor.orig/* /var/lib/linstor
# systemctl start linstor-controller
----------------------------

copy the `/etc/systemd/system/var-lib-linstor.mount` mount file to all the standby nodes for the linstor controller.
Again, do not `systemctl enable` any of these services, they get managed by `drbd-reactor`.

==== Multiple LINSTOR controllers
The next step is to install LINSTOR controllers on all nodes that have access
to the `linstor_db` DRBD resource (as they need to mount the DRBD volume) and
which you want to become a possible LINSTOR controller. It is important that the controllers
are manged by `drbd-reactor`, so make sure the `linstor-controller.service` is
disabled on all nodes! To be sure, execute `systemctl disable linstor-controller`
on all cluster nodes and `systemctl stop linstor-controller` on all nodes except the one it is currently
running from the previous step. Also make sure to set `chattr +i /var/lib/linstor` on all potential controller
nodes if you use LINSTOR version equal or greater to 1.14.0.

==== Managing the services
For starting and stopping the mount service and the linstor-controller service we use `drbd-reactor`. Install this
component on all nodes that could become a LINSTOR controller and edit their `/etc/drbd-reactor.d/linstor_db.toml` configuration
file. It should contain an enabled promoter plugin section like this:

----------------------------
[[promoter]]
id = "linstor_db"
[promoter.resources.linstor_db]
start = ["var-lib-linstor.mount", "linstor-controller.service"]
----------------------------

Depending on your requirements you might also want to set an `on-stop-failure` action and set `stop-services-on-exit`.

After that restart `drbd-reactor` and enable it on all the nodes you configured it.
----------------------------
# systemctl restart drbd-reactor
# systemctl enable drbd-reactor
----------------------------

Check that there are no warnings from drbd-reactor in the logs by running `systemctl status drbd-reactor`.
As there is already an active LINSTOR controller things will just stay the way they are.
Run `drbd-reactorctl status linstor_db` to check the health of the linstor_db target unit.

The last but never the less important step is to configure the LINSTOR
satellite services to not delete (and then regenerate) the resource file for the
LINSTOR controller DB at its startup. Do not edit the service files directly, but use `systemctl edit`. Edit
the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites.

--------------
# systemctl edit linstor-satellite
[Service]
Environment=LS_KEEP_RES=linstor_db
--------------

After this change you should execute `systemctl restart linstor-satellite` on all satellite nodes.

CAUTION: Be sure to configure your LINSTOR client for use with multiple controllers as described in
the section titled, <<s-using_the_linstor_client>> and make sure you also
configured your integration plugins (e.g., the Proxmox plugin) to be ready for
multiple LINSTOR controllers.

[[s-drbd_clients]]
=== DRBD clients
By using the `--drbd-diskless` option instead of `--storage-pool` you can
have a permanently diskless DRBD device on a node. This means that
the resource will appear as block device and can be mounted to the
filesystem without an existing storage-device. The data of the
resource is accessed over the network on another nodes with the
same resource.

----------------------------
# linstor resource create delta backups --drbd-diskless
----------------------------

NOTE: The option `--diskless` was deprecated. Please use `--drbd-diskless`
or `--nvme-initiator` instead.

[[s-linstor-drbd-consistency-group-multiple-volumes]]
=== LINSTOR - DRBD consistency group/multiple volumes

The so called consistency group is a feature from DRBD. It is mentioned in this user-guide, due to the
fact that one of LINSTOR's main functions is to manage storage-clusters with DRBD. Multiple volumes in
one resource are a consistency group.

This means that changes on different volumes from one resource are getting replicated in
the same chronological order on the other Satellites.

Therefore you don't have to worry about the timing if you have interdependent data on different volumes in a
resource.

To deploy more than one volume in a LINSTOR-resource you have to create two volume-definitions with the same name.

----
# linstor volume-definition create backups 500G
# linstor volume-definition create backups 100G
----


[[s-volumes_of_one_resource_to_different_storage_pools]]
=== Volumes of one resource to different Storage-Pools
This can be achieved by setting the `StorPoolName` property to the volume
definitions before the resource is deployed to the nodes:


----------------------------
# linstor resource-definition create backups
# linstor volume-definition create backups 500G
# linstor volume-definition create backups 100G
# linstor volume-definition set-property backups 0 StorPoolName pool_hdd
# linstor volume-definition set-property backups 1 StorPoolName pool_ssd
# linstor resource create alpha backups
# linstor resource create bravo backups
# linstor resource create charlie backups
----------------------------

NOTE: Since the `volume-definition create` command is used without the `--vlmnr` option
LINSTOR assigned the volume numbers starting at 0. In the following two
lines the 0 and 1 refer to these automatically assigned volume numbers.

Here the 'resource create' commands do not need a `--storage-pool` option.
In this case LINSTOR uses a 'fallback' storage pool. Finding that
storage pool, LINSTOR queries the properties of the following objects
in the following order:

  * Volume definition
  * Resource
  * Resource definition
  * Node

If none of those objects contain a `StorPoolName` property, the controller
falls back to a hard-coded 'DfltStorPool' string as a storage pool.

This also means that if you forgot to define a storage pool prior deploying a
resource, you will get an error message that LINSTOR could not find the
storage pool named 'DfltStorPool'.


[[s-linstor-without-drbd]]
=== LINSTOR without DRBD

LINSTOR can be used without DRBD as well. Without DRBD, LINSTOR is
able to provision volumes from LVM and ZFS backed storage pools, and
create those volumes on individual nodes in your LINSTOR cluster.

Currently LINSTOR supports the creation of LVM and ZFS
volumes with the option of layering some combinations of LUKS,
DRBD, and/or NVMe-oF/NVMe-TCP on top of those volumes.

For example, assume we have a Thin LVM backed storage pool defined in
our LINSTOR cluster named, `thin-lvm`:

----
# linstor --no-utf8 storage-pool list
+--------------------------------------------------------------+
| StoragePool | Node      | Driver   | PoolName          | ... |
|--------------------------------------------------------------|
| thin-lvm    | linstor-a | LVM_THIN | drbdpool/thinpool | ... |
| thin-lvm    | linstor-b | LVM_THIN | drbdpool/thinpool | ... |
| thin-lvm    | linstor-c | LVM_THIN | drbdpool/thinpool | ... |
| thin-lvm    | linstor-d | LVM_THIN | drbdpool/thinpool | ... |
+--------------------------------------------------------------+
----

We could use LINSTOR to create a Thin LVM on `linstor-d` that's 100GiB
in size using the following commands:

----
# linstor resource-definition create rsc-1
# linstor volume-definition create rsc-1 100GiB
# linstor resource create --layer-list storage \
          --storage-pool thin-lvm linstor-d rsc-1
----

You should then see you have a new Thin LVM on `linstor-d`. You can
extract the device path from LINSTOR by listing your linstor resources
with the `--machine-readable` flag set:

----
# linstor --machine-readable resource list | grep device_path
            "device_path": "/dev/drbdpool/rsc-1_00000",
----

If you wanted to layer DRBD on top of this volume, which is the default
`--layer-list` option in LINSTOR for ZFS or LVM backed volumes, you
would use the following resource creation pattern instead:

----
# linstor resource-definition create rsc-1
# linstor volume-definition create rsc-1 100GiB
# linstor resource create --layer-list drbd,storage \
          --storage-pool thin-lvm linstor-d rsc-1
----

You would then see that you have a new Thin LVM backing a DRBD volume
on `linstor-d`:

----
# linstor --machine-readable resource list | grep -e device_path -e backing_disk
            "device_path": "/dev/drbd1000",
            "backing_disk": "/dev/drbdpool/rsc-1_00000",
----

The following table shows which layer can be followed by which child-layer:

[cols=">1,<5"]
|===
| Layer | Child layer

| DRBD | CACHE, WRITECACHE, NVME, LUKS, STORAGE
| CACHE | WRITECACHE, NVME, LUKS, STORAGE
| WRITECACHE | CACHE, NVME, LUKS, STORAGE
| NVME | CACHE, WRITECACHE, LUKS, STORAGE
| LUKS | STORAGE
| STORAGE | -
|===

NOTE: One layer can only occur once in the layer-list

TIP: For information about the prerequisites for the `LUKS` layer,
refer to the Encrypted Volumes section of this User's Guide.

[[s-nvme-layer]]
==== NVMe-oF/NVMe-TCP LINSTOR Layer

NVMe-oF/NVMe-TCP allows LINSTOR to connect diskless resources to a
node with the same resource where the data is stored over NVMe
fabrics. This leads to the advantage that resources can be mounted
without using local storage by accessing the data over the network.
LINSTOR is not using DRBD in this case, and therefore NVMe resources
provisioned by LINSTOR are not replicated, the data is stored on one
node.

NOTE: NVMe-oF only works on RDMA-capable networks and NVMe-TCP on
every network that can carry IP traffic. If you want to know more
about NVMe-oF/NVMe-TCP visit
https://www.linbit.com/en/nvme-linstor-swordfish/ for more
information.

To use NVMe-oF/NVMe-TCP with LINSTOR the package `nvme-cli` needs to
be installed on every Node which acts as a Satellite and will use
NVMe-oF/NVMe-TCP for a resource:

IMPORTANT: If you are not using Ubuntu use the suitable command for
installing packages on your OS - SLES: zypper - CentOS: yum

------
# apt install nvme-cli
------

To make a resource which uses NVMe-oF/NVMe-TCP an additional parameter
has to be given as you create the resource-definition:

------
# linstor resource-definition create nvmedata -l nvme,storage
------

NOTE: As default the -l (layer-stack) parameter is set to `drbd,
storage` when DRBD is used. If you want to create LINSTOR resources
with neither NVMe nor DRBD you have to set the `-l` parameter to only
`storage`.

In order to use NVMe-TCP instead of the default NVMe-oF the following
property needs to be set:

------
# linstor resource-definition set-property nvmedata NVMe/TRType tcp
------

The property `NVMe/TRType` can alternatively be set on resource-group
or controller level.

Next, create the volume-definition for our resource:

------
# linstor volume-definition create nvmedata 500G
------

Before you create the resource on your nodes you have to know where
the data will be stored locally and which node accesses it over the
network.

First we create the resource on the node where our data will be stored:

------
# linstor resource create alpha nvmedata --storage-pool pool_ssd
------

On the nodes where the resource-data will be accessed over the
network, the resource has to be defined as diskless:

-----
# linstor resource create beta nvmedata --nvme-initiator
-----

Now you can  mount the resource `nvmedata` on one of your nodes.

IMPORTANT: If your nodes have more than one NIC you should force the
route between them for NVMe-of/NVME-TCP, otherwise multiple NICs
could cause troubles.

[[s-openflex]]
==== OpenFlex(TM) Layer

Since version 1.5.0 the additional Layer `openflex` can be used in LINSTOR.
From LINSTOR's perspective, the
https://www.westerndigital.com/products/storage-platforms/openflex-composable-infrastructure[OpenFlex
Composable Infrastructure] takes the role of a combined layer acting as a
storage layer (like LVM) and also providing the allocated space as an NVMe target.
OpenFlex has a REST API which is also used by LINSTOR to operate with.

As OpenFlex combines concepts of LINSTOR's storage as well as NVMe-layer, LINSTOR was
added both, a new storage driver for the storage pools as well as a dedicated `openflex`
layer which uses the mentioned REST API.

In order for LINSTOR to communicate with the OpenFlex-API, LINSTOR needs some additional
properties, which can be set once on `controller` level to take LINSTOR-cluster wide effect:

* `StorDriver/Openflex/ApiHost` specifies the host or IP of the API entry-point
* `StorDriver/Openflex/ApiPort` this property is glued with a colon to the previous to form the
basic `http://ip:port` part used by the REST calls
* `StorDriver/Openflex/UserName` the REST username
* `StorDriver/Openflex/UserPassword` the password for the REST user

Once that is configured, we can now create LINSTOR objects to represent the OpenFlex architecture.
The theoretical mapping of LINSTOR objects to OpenFlex objects are as follows:
Obviously an OpenFlex storage pool is represented by a LINSTOR storage pool. As the next thing above
a LINSTOR storage pool is already the node, a LINSTOR node represents an OpenFlex storage device.
The OpenFlex objects above storage device are not mapped by LINSTOR.

When using NVMe, LINSTOR was designed to run on both sides, the NVMe target as well as on the
NVMe initiator side. In the case of OpenFlex, LINSTOR cannot (or even should not) run on the NVMe
target side as that is completely managed by OpenFlex. As LINSTOR still needs nodes and storage pools
to represent the OpenFlex counterparts, the LINSTOR client was extended with special node create commands
since 1.0.14. These commands not only accept additionally needed configuration data, but also
starts a "special satellite" besides the already running controller instance. This special satellites
are completely LINSTOR managed, they will shutdown when the controller shuts down and will be started
again when the controller starts.
The new client command for creating a "special satellite" representing an OpenFlex storage device is:

----
$ linstor node create-openflex-target ofNode1 192.168.166.7 000af795789d
----
The arguments are as follows:

* `ofNode1` is the node name which is also used by the standard `linstor node create` command
* `192.168.166.7` is the address on which the provided NVMe devices can be accessed.
As the NVMe devices are accessed by a dedicated network interface, this address differs from the
address specified with the property `StorDriver/Openflex/ApiHost`. The latter is used for the
management / REST API.
* `000af795789d` is the identifier for the OpenFlex storage device.

The last step of the configuration is the creation of LINSTOR storage pools:

----
$ linstor storage-pool create openflex ofNode1 sp0 0
----

* `ofNode1` and `sp0` are the node name and storage pool name, respectively, just as usual for the
LINSTOR's create storage pool command
* The last `0` is the identifier of the OpenFlex storage pool within the previously defined storage device

Once all necessary storage pools are created in LINSTOR, the next steps are similar to
the usage of using an NVMe resource with LINSTOR. Here is a complete example:

----
# set the properties once
linstor controller set-property StorDriver/Openflex/ApiHost 10.43.7.185
linstor controller set-property StorDriver/Openflex/ApiPort 80
linstor controller set-property StorDriver/Openflex/UserName myusername
linstor controller set-property StorDriver/Openflex/UserPassword mypassword

# create a node for openflex storage device "000af795789d"
linstor node create-openflex-target ofNode1 192.168.166.7 000af795789d

# create a usual linstor satellite. later used as nvme initiator
linstor node create bravo

# create a storage pool for openflex storage pool "0" within storage device "000af795789d"
linstor storage-pool create openflex ofNode1 sp0 0

# create resource- and volume-definition
linstor resource-definition create backupRsc
linstor volume-definition create backupRsc 10G

# create openflex-based nvme target
linstor resource create ofNode1 backupRsc --storage-pool sp0 --layer-list openflex

# create openflex-based nvme initiator
linstor resource create bravo backupRsc --nvme-initiator --layer-list openflex
----

NOTE: In case a node should access the OpenFlex REST API through a different host than specified with +
`linstor controller set-property StorDriver/Openflex/ApiHost 10.43.7.185` you can always use LINSTOR's
inheritance mechanism for properties. That means simply define the same property on the node-level
you need it, i.e. +
`linstor node set-property ofNode1 StorDriver/Openflex/ApiHost 10.43.8.185`

[[s-writecache-layer]]
==== Writecache Layer

A https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/writecache.html[DM-Writecache]
device is composed by two devices, one storage device and one cache device.
LINSTOR can setup such a writecache device, but needs some additional information, like
the storage pool and the size of the cache device.

------
# linstor storage-pool create lvm node1 lvmpool drbdpool
# linstor storage-pool create lvm node1 pmempool pmempool

# linstor resource-definition create r1
# linstor volume-definition create r1 100G

# linstor volume-definition set-property r1 0 Writecache/PoolName pmempool
# linstor volume-definition set-property r1 0 Writecache/Size 1%

# linstor resource create node1 r1 --storage-pool lvmpool --layer-list WRITECACHE,STORAGE
------

The two properties set in the examples are mandatory, but can also be set on
controller level which would act as a default for all resources with `WRITECACHE` in their
`--layer-list`. However, please note that the `Writecache/PoolName` refers to
the corresponding node. If the node does not have a storage-pool named `pmempool` you will
get an error message.

The 4 mandatory parameters required by
https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/writecache.html[DM-Writecache]
are either configured via property or figured out by LINSTOR.
The optional properties listed in the mentioned link can also be set via property.
Please see `linstor controller set-property --help` for a list of `Writecache/*`
property-keys.

Using `--layer-list DRBD,WRITECACHE,STORAGE` while having DRBD configured to use
external metadata, only the backing device will use a writecache, not the
device holding the external metadata.

[[s-cache-layer]]
==== Cache Layer

LINSTOR can also setup a https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/cache.html[DM-Cache]
device, which is very similar to the DM-Writecache from the previous section. The major difference
is that a cache device is composed by three devices: one storage device, one cache device and one
meta device. The LINSTOR properties are quite similar to those of the writecache but are located
in the `Cache` namespace:

------
# linstor storage-pool create lvm node1 lvmpool drbdpool
# linstor storage-pool create lvm node1 pmempool pmempool

# linstor resource-definition create r1
# linstor volume-definition create r1 100G

# linstor volume-definition set-property r1 0 Cache/CachePool pmempool
# linstor volume-definition set-property r1 0 Cache/Cachesize 1%

# linstor resource create node1 r1 --storage-pool lvmpool --layer-list CACHE,STORAGE
------

NOTE: Instead of `Writecache/PoolName` (as when configuring the Writecache layer) the
Cache layer's only required property is called `Cache/CachePool`. The reason for this
is that the Cache layer also has a `Cache/MetaPool` which can be configured separately
or it defaults to the value of `Cache/CachePool`.

Please see `linstor controller set-property --help` for a list of `Cache/*`
property-keys and default values for omitted properties.

Using `--layer-list DRBD,CACHE,STORAGE` while having DRBD configured to use external
metadata, only the backing device will use a cache, not the device holding the external
metadata.

[[s-storage-layer]]
==== Storage Layer

The storage layer will provide new devices from well known volume managers like LVM, ZFS or
others. Every layer combination needs to be based on a storage layer, even if the resource
should be diskless - for that type there is a dedicated `diskless` provider type.

For a list of providers with their properties please see <<s-storage-proviers, Storage Providers>>.

For some storage providers LINSTOR has special properties:

* `StorDriver/WaitTimeoutAfterCreate`: If LINSTOR expects a device to appear after creation
(for example after calls of `lvcreate`, `zfs create`,...), LINSTOR waits per default 500ms
for the device to appear. These 500ms can be overridden by this property.
* `StorDriver/dm_stats`: If set to `true` LINSTOR calls `dmstats create $device` after
creation and `dmstats delete $device --allregions` after deletion of a volume.
Currently only enabled for LVM and LVM_THIN storage providers.

[[s-storage-providers]]
=== Storage Providers

LINSTOR has a few storage providers. The most used ones are LVM and ZFS. But also for those
two providers there are already sub-types for their thinly provisioned variants.

* Diskless: This provider type is mostly required to have a storage pool that can be configured
with LINSTOR properties like `PrefNic` as described in <<s-managing_network_interface_cards,
Managing Network Interface Cards>>.
* LVM / LVM-Thin: The adminstrator is expected to specify the LVM volume group or the thin-pool
(in form of "LV/thinpool") in order to use the corresponding storage type. These drivers support
following properties for fine-tuning:
** `StorDriver/LvcreateOptions`: The value of this property is appended to every
  `lvcreate ...` call LINSTOR executes.
* ZFS / ZFS-Thin: The administrator is expected to specify the ZPool that LINSTOR should use.
These drivers support following properties for fine-tuning:
** `StorDriver/ZfscreateOptions`: The value of this property is appended to every
  `zfs create ...` call LINSTOR executes.
* File / FileThin: Mostly used for demonstration / experiments. LINSTOR will basically reserve a
file in a given directory and will configure a https://man7.org/linux/man-pages/man4/loop.4.html[loop device]
on top of that file.
* OpenFlex: This special storage provider currently requires to be run on a "special satellite".
Please see <<s-openflex, OpenFlex(TM) Layer>> for more details.
* EXOS: This special storage provider currently requires to be run on a "special satellite".
Please see the <<ch-exos, EXOS Integration>> chapter
* SPDK: The administrator is expected to speicify the logical volume store which LINSTOR should use.
The usage of this storage provider implies the usage of the <<s-nvme-layer,NVME Layer>>.
** Remote-SPDK: This special storage provider currently requires to be run on a "special satellite".
Please see <<s-remote-spdk, Remote SPDK Provider>> for more details.

[[s-remote-spdk]]
==== Remote SPDK Provider
A storage pool with the type remote SPDK can only be created on a "special satellite". For this
you first need to start a new satellite using the command:

----
$ linstor node create-remote-spdk-target nodeName 192.168.1.110
----

This will start a new satellite instance running on the same machine as the controller. This special
satellite will do all the REST based RPC communication towards the remote SPDK proxy. As the help message
of the LINSTOR command shows, the administrator might want to use additional settings when creating
this special satellite:

----
$ linstor node create-remote-spdk-target -h
usage: linstor node create-remote-spdk-target [-h] [--api-port API_PORT]
                                              [--api-user API_USER]
                                              [--api-user-env API_USER_ENV]
                                              [--api-pw [API_PW]]
                                              [--api-pw-env API_PW_ENV]
                                              node_name api_host
----

The difference between the `--api-\*` and their corresponding `--api-\*-env` versions is that the
version with the `-env` ending will look for an environment variable containing the actual value to use
whereas the `--api-\*` version directly take the value which is stored in the LINSTOR property.
Administrators might not want to save the `--api-pw` in plaintext, which would be clearly visible
using commands like `linstor node list-property <nodeName>`.

Once that special satellite is up and running the actual storage pool can be created:

----
$ linstor storage-pool create remotespdk -h
usage: linstor storage-pool create remotespdk [-h]
                                              [--shared-space SHARED_SPACE]
                                              [--external-locking]
                                              node_name name driver_pool_name
----

Whereas `node_name` is self-explanatory, `name` is the name of the LINSTOR storage pool and
`driver_pool_name` refers to the SPDK logical volume store.

Once this _remotespdk_ storage pool is created the remaining procedure is quite similar as
using NVMe: First the target has to be created by creating a simple "diskful" resource followed
by a second resource having the `--nvme-initiator` option enabled.


[[s-managing_network_interface_cards]]
=== Managing Network Interface Cards

LINSTOR can deal with multiple network interface cards (NICs) in a machine,
they are called `netif` in LINSTOR speak.

NOTE: When a satellite node is created a first `netif` gets created implicitly
with the name `default`. Using the `--interface-name` option of the `node create`
command you can give it a different name.

Additional NICs are created like this:
----------------------------
# linstor node interface create alpha 100G_nic 192.168.43.221
# linstor node interface create alpha 10G_nic 192.168.43.231
----------------------------

NICs are identified by the IP address only, the name is arbitrary and is
*not* related to the interface name used by Linux. The NICs can be assigned
to storage pools so that whenever a resource is created in such a storage
pool, the DRBD traffic will be routed through the specified NIC.

----------------------------
# linstor storage-pool set-property alpha pool_hdd PrefNic 10G_nic
# linstor storage-pool set-property alpha pool_ssd PrefNic 100G_nic
----------------------------

FIXME describe how to route the controller +<->+ client communication through
a specific `netif`.

[[s-linstor-encrypted-volumes]]
=== Encrypted volumes
LINSTOR can handle transparent encryption of drbd volumes. dm-crypt is used to
encrypt the provided storage from the storage device.

NOTE: In order to use dm-crypt please make sure to have `cryptsetup` installed before
you start the satellite

Basic steps to use encryption:

1. Disable user security on the controller (this will be obsolete once authentication works)
2. Create a master passphrase
3. Add `luks` to the layer-list. Note that all plugins (e.g., Proxmox) require a DRBD layer as the top most layer if they do not explicitly state otherwise.
4. Don't forget to re-enter the master passphrase after a controller restart.

[[s-disable_user_security]]
==== Disable user security
Disabling the user security on the `Linstor` controller is a one time operation and is
afterwards persisted.

1. Stop the running linstor-controller via systemd: `systemctl stop linstor-controller`
2. Start a linstor-controller in debug mode: `/usr/share/linstor-server/bin/Controller -c /etc/linstor -d`
3. In the debug console enter: `setSecLvl secLvl(NO_SECURITY)`
4. Stop linstor-controller with the debug shutdown command: `shutdown`
5. Start the controller again with systemd: `systemctl start linstor-controller`

[[s-encrypt_commands]]
==== Encrypt commands
Below are details about the commands.

Before LINSTOR can encrypt any volume a master passphrase needs to be created.
This can be done with the linstor-client.

----
# linstor encryption create-passphrase
----

`crypt-create-passphrase` will wait for the user to input the initial master passphrase
(as all other crypt commands will with no arguments).

If you ever want to change the master passphrase this can be done with:

----
# linstor encryption modify-passphrase
----

The `luks` layer can be added when creating the resource-definition or the resource
itself, whereas the former method is recommended since it will be automatically applied
to all resource created from that resource-definition.

----
# linstor resource-definition create crypt_rsc --layer-list luks,storage
----

To enter the master passphrase (after controller restart) use the following command:

----
# linstor encryption enter-passphrase
----

NOTE: Whenever the linstor-controller is restarted, the user has to send
the master passphrase to the controller, otherwise LINSTOR is unable to reopen or
create encrypted volumes.

[[s-automatic_passphrase]]
==== Automatic Passphrase
It is possible to automate the process of creating and re-entering the master passphrase.

To use this, either an environment variable called `MASTER_PASSPHRASE` or an entry in
`/etc/linstor/linstor.toml` containing the master passphrase has to be created.

The required `linstor.toml` looks like this:

----
[encrypt]
passphrase="example"
----

If either one of these is set, then every time the controller starts it will check whether
a master passphrase already exists. If there is none, it will create a new master passphrase as specified.
Otherwise, the controller enters the passphrase.

WARNING: If a master passphrase is already configured, and it is not the same one as specified
in the environment variable or `linstor.toml`, the controller will be unable to re-enter the
master passphrase and react as if the user had entered a wrong passphrase.
This can only be resolved through manual input from the user, using the same commands as if
the controller was started without the automatic passphrase.

NOTE: In case the master passphrase is set in both an environment variable and the `linstor.toml`,
only the master passphrase from the `linstor.toml` will be used.

[[s-linstor-status]]
=== Checking the state of your cluster
LINSTOR provides various commands to check the state of your cluster.
These commands start with a 'list-' prefix and provide various filtering and
sorting options. The '--groupby' option can be used to group and sort the
output in multiple dimensions.

----------------------------
# linstor node list
# linstor storage-pool list --groupby Size
----------------------------

[[s-linstor-snapshots]]
=== Managing snapshots
Snapshots are supported with thin LVM and ZFS storage pools.

[[s-creating_a_snapshot-linstor]]
==== Creating a snapshot
Assuming a resource definition named 'resource1' which has been placed on some
nodes, a snapshot can be created as follows:

----------------------------
# linstor snapshot create resource1 snap1
----------------------------

This will create snapshots on all nodes where the resource is present.
LINSTOR will ensure that consistent snapshots are taken even when the
resource is in active use.

Setting the resource-definition property `AutoSnapshot/RunEvery`
LINSTOR will automatically create snapshots every X minute.
The optional property `AutoSnapshot/Keep` can be used to clean-up old snapshots
which were created automatically. No manually created snapshot will be cleaned-up / deleted.
If `AutoSnapshot/Keep` is omitted (or <= 0), LINSTOR will keep the last 10 snapshots
by default.

----------------------------
# linstor resource-definition set-property AutoSnapshot/RunEvery 15
# linstor resource-definition set-property AutoSnapshot/Keep 5
----------------------------


[[s-restoring_a_snapshot-linstor]]
==== Restoring a snapshot
The following steps restore a snapshot to a new resource.
This is possible even when the original resource has been removed
from the nodes where the snapshots were taken.

First define the new resource with volumes matching those from the snapshot:

----------------------------
# linstor resource-definition create resource2
# linstor snapshot volume-definition restore --from-resource resource1 --from-snapshot snap1 --to-resource resource2
----------------------------

At this point, additional configuration can be applied if necessary.
Then, when ready, create resources based on the snapshots:

----------------------------
# linstor snapshot resource restore --from-resource resource1 --from-snapshot snap1 --to-resource resource2
----------------------------

This will place the new resource on all nodes where the snapshot is present.
The nodes on which to place the resource can also be selected explicitly;
see the help (`linstor snapshot resource restore -h`).

[[s-rolling_back_snapshot-linstor]]
==== Rolling back to a snapshot
LINSTOR can roll a resource back to a snapshot state.
The resource must not be in use.
That is, it may not be mounted on any nodes.
If the resource is in use, consider whether you can achieve your goal by
<<s-restoring_a_snapshot-linstor,restoring the snapshot>> instead.

Rollback is performed as follows:

----------------------------
# linstor snapshot rollback resource1 snap1
----------------------------

A resource can only be rolled back to the most recent snapshot.
To roll back to an older snapshot, first delete the intermediate snapshots.

[[s-removing_a_snapshot-linstor]]
==== Removing a snapshot
An existing snapshot can be removed as follows:

----------------------------
# linstor snapshot delete resource1 snap1
----------------------------

[[s-shipping_snapshots-linstor]]
==== Shipping a snapshot

Snapshots can be shipped between LINSTOR nodes or between different LINSTOR clusters,
as well as to an S3 storage such as https://aws.amazon.com/s3/[Amazon S3] or https://min.io/[min.io].

[[s-shipping_snapshots-remotes]]
===== Remotes

In a LINSTOR cluster, the definition of a shipping target is called a remote.
Currently, there are two different types of remotes: LINSTOR remotes and S3 remotes.
LINSTOR remotes are used to ship snapshots to a different LINSTOR cluster, while
S3 remotes are needed to ship snapshots to AWS S3, min.io or any other service using S3
compatible object storage.

IMPORTANT: Since a remote needs to store sensitive data, such as passwords, it is
neccessary to have encryption enabled whenever you wish to use a remote in any way.
How to set up LINSTOR's encryption is described <<s-encrypt_commands,here>>.

To create an S3 remote, LINSTOR will need to know the endpoint
(that is, the URL of the target S3 server), the name of the target bucket, the region the S3
server is in, as well as the access-key and secret-key used to access the bucket. If the command
is sent without adding the secret-key, a prompt will pop up to enter it in. The command
should look like this:

--------------------------
# linstor remote create s3 myRemote s3.us-west-2.amazonaws.com my-bucket us-west-2 admin password
--------------------------

TIP: Usually, LINSTOR uses the endpoint and bucket to create an url using the virtual-hosted-style
for its access to the given bucket (e.g. my-bucket.s3.us-west-2.amazonaws.com). Should your setup not
allow access this way, change the remote to path-style access (e.g. s3.us-west-2.amazonaws.com/my-bucket)
by adding the `--use-path-style` argument to make LINSTOR combine the parameters accordingly.

To create a LINSTOR remote, only the URL or IP address of the controller of the target cluster
is needed. The command goes as follows:

----------------------------
# linstor remote create linstor myRemote 192.168.0.15
----------------------------

Additionally, to ship LUKS-based (encrypted) backups, it is necessary to add the `--passphrase`
and `--cluster-id` arguments to the command. This is used to save the passphrase and cluster-id of
the target cluster to the remote respectively. For more details on shipping LUKS-based backups
between two LINSTOR clusters, see <<s-shipping_snapshots-l2l,this chapter>>.

To see all the remotes known to the local cluster, use `linstor remote list`. To delete a remote, use
`linstor remote delete myRemoteName`. Should an existing remote need altering, use `linstor remote
modify` to change it.

[[s-shipping_snapshots-s3]]
===== Shipping snapshots to S3

All that is needed to ship a snapshot to S3 is to create an S3-remote that the current cluster can reach
as well as the resource that should be shipped. Then, simply use the following command to ship it there:

--------------------------
# linstor backup create myRemote myRsc
--------------------------

This command will create a snapshot of your resource and ship it to the given remote. If this
isn't the first time you shipped a backup of this resource (to that remote) and the snapshot
of the previous backup hasn't been deleted yet, an incremental backup will be shipped.
To force the creation of a full backup, add the `--full` argument to the command. Getting a
specific node to ship the backup is also possible by using `--node myNode`, but if the specified
node is not available or only has the resource diskless, a different node will be chosen.

To see which backups exist in a specific remote, use `linstor backup list myRemote`. A resource-name
can be added to the command as a filter to only show backups of that specific resource by using the
argument `--resource myRsc`. If you use the `--other` argument, only entries in the bucket that LINSTOR
does not recognize as a backup will be shown. LINSTOR always names backups in a certain way, and
as long as an item in the remote is named according to this schema, it is assumed that it is a backup
created by LINSTOR - so this list will show everything else.

There are several options when it comes to deleting backups:

* `linstor backup delete all myRemote`: This command deletes ALL S3-objects on the given
remote, as long as they are recognized to be backups, that is, fit the expected naming schema.
There is the option `--cluster` to only delete backups that were created by the current cluster.

* `linstor backup delete id myRemote my-rsc_back_20210824_072543`: This command deletes
a single backup from the given remote - namely the one with the given id, which consists
of the resource-name, the automatically generated snapshot-name (back_timestamp) and, if
set, the backup-suffix. The option `--prefix` lets you delete all backups starting with
the given id. The option `--cascade` deletes not only the specified backup, but all other
incremental backups depending on it.

* `linstor backup delete filter myRemote ...`: This command has a few different arguments
to specify a selection of backups to delete. `-t 20210914_120000` will delete all backups
made before 12 o'clock on the 14th of September, 2021. `-n myNode` will delete all backups
uploaded by the given node. `-r myRsc` will delete all backups with the given resource name.
These filters can be combined as needed. Finally, `--cascade` deletes not only the selected
backup(s), but all other incremental backups depending on any of the selected backups.

* `linstor backup delete s3key myRemote randomPictureInWrongBucket`: This command will find
the object with the given S3-key and delete it - without considering anything else. This should
only be used to either delete non-backup items from the remote, or to clean up a broken backup
that is no longer deleteable by other means. Using this command to delete a regular, working
backup will break that backup, so beware!

WARNING: All commands that have the `--cascade` option will NOT delete a backup that has
incremental backups depending on it unless you explicitly add that option.

TIP: All `linstor backup delete ...` commands have the `--dry-run` option, which will
give you a list of all the S3-objects that will be deleted. This can be used to ensure
nothing that should not be deleted is accidentally deleted.

Maybe the most important task after creating a backup is restoring it. To do so, only the remote
is needed - but it is also possible to restore into an existing resource definition with no existing
snapshots nor resources. There are two options for the command:

-----------------------
# linstor backup restore myRemote myNode targetRsc --resource sourceRsc
# linstor backup restore myRemote myNode targetRsc --id sourceRsc_back_20210824_072543
-----------------------

Either `--resource (-r)` or `--id` must be used, but you cannot use both of them together. `-r` is used to
restore the latest backup of the resource specified with this option, while `--id` restores the
exact backup specified by the given id, and can therefore be used to restore backups other than
the most recent.

If the backup to be restored includes a LUKS layer, the `--passphrase` argument is required. With
it, the passphrase of the original cluster of the backup needs to be set so that LINSTOR can decrypt
the volumes after download and re-encrypt them with the local passphrase.

The backup restore will download all the snapshots from the last full backup up to the specified
backup. Afterwards, it restores the snapshots into a new resource. If that last step should be skipped,
the `--download-only` option needs to be added to the command.

Backups can be downloaded from any cluster, not just the one that uploaded them, as long as the setup
is correct. Specifically, the target resource cannot have any existing resources or snapshots, and the
storage pool(s) used need to have the same storage providers. If the storage pool(s) on the target
node have the exact same names as on the cluster the backup was created on, no extra action is
necessary. Should they have different names, the option `--storpool-rename` needs to be used. It
expects at least one `oldname=newname` pair. For every storage pool of the original backup that
is not named in that list, it will be assumed that its name is exactly the same on the target node.

To find out exactly which storage pools need to be renamed, as well as how big the download and the
restored resource will be, the command `linstor backup info myRemote ...` can be used. Similar to the restore
command, either `-r` or `--id` need to be given, which add the same restrictions as with that command.
To see how much space will be left over in the local storage pools after a restore, the argument `-n myNode`
needs to be added. Just like with a restore, it assumes the storage pool names are exactly the same
on the given node as with the backup. Should that not be the case, again, just like with the restore
command, `--storpool-rename` should be used.

[[s-shipping_snapshots-l2l]]
===== Shipping snapshots whithin LINSTOR

Shipping a snapshot directly between two LINSTOR clusters can be done with a linstor-remote as well
as a resource definition with at least one diskful resource on the source side (where the command
is issued). On the target side, you only need a linstor-remote with the cluster-id of the source-cluster set.
If you wish to ship a snapshot inside the same cluster just use a remote pointing to the local controller.
The command looks like this:

--------------------------
# linstor backup ship myRemote localRsc targetRsc
--------------------------

Additionally, you can use `--source-node` and `--target-node` to specify which node should send and
receive the backup respectively. In case those nodes are not available, a different one will be chosen
automatically.

If the target resource already has resources deployed or the `--download-only` option was specified,
the snapshots will only be shipped to the target cluster, but not restored.

When the snapshot you wish to ship contains a LUKS layer, the remote on the target cluster also needs
the passphrase of the source cluster set.

[[s-shipping_snapshots-old]]
===== Shipping a snapshot in the same cluster

WARNING: The `snapshot ship` command is considered deprecated and any bugs found
with it will not be fixed. Instead, use the `backup ship` command with a remote pointing
to your local controller. For more details, see <<s-shipping_snapshots-l2l,the previous section>>.

Both, the source as well as the target node have to have the resource for
snapshot shipping deployed. Additionally, the target resource has to be
deactivated.

----------------------------
# linstor resource deactivate nodeTarget resource1
----------------------------

WARNING: Deactivating a resource with DRBD in its layer-list can NOT be
reactivated again. However, a successfully shipped snapshot of a DRBD resource
can still be <<s-restoring_a_snapshot-linstor,restored into a new resource>>.

To manually start the snapshot-shipping, use:

----------------------------
# linstor snapshot ship --from-node nodeSource --to-node nodeTarget --resource resource1
----------------------------

By default, the snapshot-shipping uses tcp ports from the range 12000-12999. To change
this range, the property `SnapshotShipping/TcpPortRange`, which accepts a to-from range,
can be set on the controller:

----------------------------
# linstor controller set-property SnapshotShipping/TcpPortRange 10000-12000
----------------------------

A resource can also be periodically shipped. To accomplish this, it is mandatory to
set the properties `SnapshotShipping/TargetNode` as well as `SnapshotShipping/RunEvery`
on the resource-definition.
`SnapshotShipping/SourceNode` can also be set, but if omitted LINSTOR will choose
an active resource of the same resource-definition.

To allow incremental snapshot-shipping, LINSTOR has to keep at least the last shipped
snapshot on the target node. The property `SnapshotShipping/Keep` can be used to specify
how many snapshots LINSTOR should keep. If the property is not set (or <= 0) LINSTOR
will keep the last 10 shipped snapshots by default.

----------------------------
# linstor resource-definition set-property resource1 SnapshotShipping/TargetNode nodeTarget
# linstor resource-definition set-property resource1 SnapshotShipping/SourceNode nodeSource
# linstor resource-definition set-property resource1 SnapshotShipping/RunEvery 15
# linstor resource-definition set-property resource1 SnapshotShipping/Keep 5
----------------------------

[[s-linstor-setupopts]]
=== Setting options for resources

DRBD options are set using LINSTOR commands.
Configuration in files such as `/etc/drbd.d/global_common.conf` that are not
managed by LINSTOR will be ignored.
The following commands show the usage and available options:

----------------------------
# linstor controller drbd-options -h
# linstor resource-definition drbd-options -h
# linstor volume-definition drbd-options -h
# linstor resource drbd-peer-options -h
----------------------------

For instance, it is easy to set the DRBD protocol for a resource named
`backups`:

----------------------------
# linstor resource-definition drbd-options --protocol C backups
----------------------------

[[s-linstor-toggle-disk]]
=== Adding and removing disks
LINSTOR can convert resources between diskless and having a disk.
This is achieved with the `resource toggle-disk` command,
which has syntax similar to `resource create`.

For instance, add a disk to the diskless resource `backups` on 'alpha':

----------------------------
# linstor resource toggle-disk alpha backups --storage-pool pool_ssd
----------------------------

Remove this disk again:

----------------------------
# linstor resource toggle-disk alpha backups --diskless
----------------------------

[[s-linstor-migrate-disk]]
==== Migrating disks
In order to move a resource between nodes without reducing redundancy at any point,
LINSTOR's disk migrate feature can be used.
First create a diskless resource on the target node,
and then add a disk using the `--migrate-from` option.
This will wait until the data has been synced to the new disk and then remove
the source disk.

For example, to migrate a resource `backups` from 'alpha' to 'bravo':

----------------------------
# linstor resource create bravo backups --drbd-diskless
# linstor resource toggle-disk bravo backups --storage-pool pool_ssd --migrate-from alpha
----------------------------

[[s-linstor-proxy]]
=== DRBD Proxy with LINSTOR

LINSTOR expects DRBD Proxy to be running on the nodes which are involved in the
relevant connections. It does not currently support connections via DRBD Proxy
on a separate node.

Suppose our cluster consists of nodes 'alpha' and 'bravo' in a local network
and 'charlie' at a remote site, with a resource definition named `backups`
deployed to each of the nodes. Then DRBD Proxy can be enabled for the
connections to 'charlie' as follows:

----------------------------
# linstor drbd-proxy enable alpha charlie backups
# linstor drbd-proxy enable bravo charlie backups
----------------------------

The DRBD Proxy configuration can be tailored with commands such as:

----------------------------
# linstor drbd-proxy options backups --memlimit 100000000
# linstor drbd-proxy compression zlib backups --level 9
----------------------------

LINSTOR does not automatically optimize the DRBD configuration for
long-distance replication, so you will probably want to set some configuration
options such as the protocol:

----------------------------
# linstor resource-connection drbd-options alpha charlie backups --protocol A
# linstor resource-connection drbd-options bravo charlie backups --protocol A
----------------------------

Please contact LINBIT for assistance optimizing your configuration.

==== Automatically enable DRBD Proxy

LINSTOR can also be configured to automatically enable the above mentioned Proxy
connection between two nodes. For this automation, LINSTOR first needs to know
on which site each node is.

----------------------------
# linstor node set-property alpha Site A
# linstor node set-property bravo Site A
# linstor node set-property charlie Site B
----------------------------

As the `Site` property might also be used for other site-based decisions in
future features, the `DrbdProxy/AutoEnable` also has to be set to `true`:

----------------------------
# linstor controller set-property DrbdProxy/AutoEnable true
----------------------------

This property can also be set on node, resource-definition, resource and
resource-connection level (from left to right in increasing priority, whereas
the controller is the left-most, i.e. least prioritized level)

Once this initialization steps are completed, every newly created resource
will automatically check if it has to enable DRBD proxy to any of its
peer-resources.


[[s-linstor-external-database]]
=== External database

It is possible to have LINSTOR working with an external database provider
like PostgreSQL, MariaDB and since version 1.1.0 even ETCD key value store is supported.

To use an external database there are a few additional steps to configure.
You have to create a DB/Schema and user to use for linstor, and configure this in the
`/etc/linstor/linstor.toml`.


[[s-postgresql]]
==== PostgreSQL

A sample PostgreSQL `linstor.toml` looks like this:

------------------------------------------------------
[db]
user = "linstor"
password = "linstor"
connection_url = "jdbc:postgresql://localhost/linstor"
------------------------------------------------------

[[s-mariadb_mysql]]
==== MariaDB/MySQL

A sample MariaDB `linstor.toml` looks like this:

------------------------------------------------------
[db]
user = "linstor"
password = "linstor"
connection_url = "jdbc:mariadb://localhost/LINSTOR?createDatabaseIfNotExist=true"
------------------------------------------------------

NOTE: The LINSTOR schema/database is created as `LINSTOR` so make sure the MariaDB connection string
refers to the `LINSTOR` schema, as in the example above.


[[s-etcd]]
==== ETCD

ETCD is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup.
The ETCD driver is already included in the LINSTOR-controller package and only needs to be configured in the `linstor.toml`.

More information on how to install and configure ETCD can be found here: https://etcd.io/docs[ETCD docs]

And here is a sample [db] section from the `linstor.toml`:

------------------------------------------------------
[db]
## only set user/password if you want to use authentication, only since LINSTOR 1.2.1
# user = "linstor"
# password = "linstor"

## for etcd
## do not set user field if no authentication required
connection_url = "etcd://etcdhost1:2379,etcdhost2:2379,etcdhost3:2379"

## if you want to use TLS, only since LINSTOR 1.2.1
# ca_certificate = "ca.pem"
# client_certificate = "client.pem"

## if you want to use client TLS authentication too, only since LINSTOR 1.2.1
# client_key_pkcs8_pem = "client-key.pkcs8"
## set client_key_password if private key has a password
# client_key_password = "mysecret"
------------------------------------------------------

[[s-linstor-rest-api]]
=== LINSTOR REST-API

To make LINSTOR's administrative tasks more accessible and also available for web-frontends a
REST-API has been created. The REST-API is embedded in the linstor-controller
and since LINSTOR 0.9.13 configured via the `linstor.toml` configuration file.

---------
[http]
  enabled = true
  port = 3370
  listen_addr = "127.0.0.1"  # to disable remote access
---------


If you want to use the REST-API the current documentation can be found on the following link:
https://app.swaggerhub.com/apis-docs/Linstor/Linstor/

[[s-linstor-rest-api-https]]
==== LINSTOR REST-API HTTPS

The HTTP REST-API can also run secured by HTTPS and is highly recommended if you use any features that
require authorization. Todo so you have to create a java keystore file with a valid certificate
that will be used to encrypt all HTTPS traffic.

Here is a simple example on how you can create a self signed certificate with the `keytool` that is included
in the java runtime:

---------
keytool -keyalg rsa -keysize 2048 -genkey -keystore ./keystore_linstor.jks\
 -alias linstor_controller\
 -dname "CN=localhost, OU=SecureUnit, O=ExampleOrg, L=Vienna, ST=Austria, C=AT"
---------

`keytool` will ask for a password to secure the generated keystore file and is needed for the
LINSTOR-controller configuration.
In your `linstor.toml` file you have to add the following section:

---------
[https]
  keystore = "/path/to/keystore_linstor.jks"
  keystore_password = "linstor"
---------

Now (re)start the `linstor-controller` and the HTTPS REST-API should be available on port 3371.

More information on how to import other certificates can be found here:
https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html

NOTE: When HTTPS is enabled, all requests to the HTTP /v1/ REST-API will be redirected to the
HTTPS redirect.


[[s-rest-api-https-restricted-client]]
===== LINSTOR REST-API HTTPS restricted client access

Client access can be restricted by using a SSL truststore on the Controller.
Basically you create a certificate for your client and add it to your truststore and the client
then uses this certificate for authentication.

First create a client certificate:

---------
keytool -keyalg rsa -keysize 2048 -genkey -keystore client.jks\
 -storepass linstor -keypass linstor\
 -alias client1\
 -dname "CN=Client Cert, OU=client, O=Example, L=Vienna, ST=Austria, C=AT"
---------

Then we import this certificate to our controller truststore:

---------
keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore client.jks -destkeystore trustore_client.jks
---------

And enable the truststore in the `linstor.toml` configuration file:

---------
[https]
  keystore = "/path/to/keystore_linstor.jks"
  keystore_password = "linstor"
  truststore = "/path/to/trustore_client.jks"
  truststore_password = "linstor"
---------

Now restart the Controller and it will no longer be possible to access the controller API without a
correct certificate.

The LINSTOR client needs the certificate in PEM format, so before we can use it we have to convert
the java keystore certificate to the PEM format.

---------
# Convert to pkcs12
keytool -importkeystore -srckeystore client.jks -destkeystore client.p12\
 -storepass linstor -keypass linstor\
 -srcalias client1 -srcstoretype jks -deststoretype pkcs12

# use openssl to convert to PEM
openssl pkcs12 -in client.p12 -out client_with_pass.pem
---------

To avoid entering the PEM file password all the time it might be convenient to remove the password.

---------
openssl rsa -in client_with_pass.pem -out client1.pem
openssl x509 -in client_with_pass.pem >> client1.pem
---------

Now this PEM file can easily be used in the client:

---------
linstor --certfile client1.pem node list
---------


The `--certfile` parameter can also added to the client configuration file, see
<<s-using_the_linstor_client>> for more details.


[[s-linstor-logging]]
=== Logging

Linstor uses https://www.slf4j.org/[SLF4J] with https://logback.qos.ch/[Logback] as binding. This gives
Linstor the possibility to distinguish between the log levels `ERROR`, `WARN`, `INFO`, `DEBUG` and `TRACE`
(in order of increasing verbosity). In the current linstor version (1.1.2) the user has the following
four methods to control the logging level, ordered by priority (first has highest priority):


. `TRACE` mode can be `enabled` or `disabled` using the debug console:
+
----
Command ==> SetTrcMode MODE(enabled)
SetTrcMode           Set TRACE level logging mode
New TRACE level logging mode: ENABLED
----

. When starting the controller or satellite a command line argument can be passed:
+
----
java ... com.linbit.linstor.core.Controller ... --log-level INFO
java ... com.linbit.linstor.core.Satellite  ... --log-level INFO
----

. The recommended place is the `logging` section in the configuration file.
The default configuration file location is `/etc/linstor/linstor.toml` for the controller
and `/etc/linstor/linstor_satellite.toml` for the satellite.
Configure the logging level as follows:
+
----
[logging]
   level="INFO"
----

. As Linstor is using Logback as an implementation, `/usr/share/linstor-server/lib/logback.xml` can
also be used. Currently only this approach supports different log levels for different components, like
shown in the example below:
+
----
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="false" scanPeriod="60 seconds">
<!--
 Values for scanPeriod can be specified in units of milliseconds, seconds, minutes or hours
 https://logback.qos.ch/manual/configuration.html
-->
 <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
   <!-- encoders are assigned the type
        ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
   <encoder>
     <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</pattern>
   </encoder>
 </appender>
 <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
   <file>${log.directory}/linstor-${log.module}.log</file>
   <append>true</append>
   <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
     <Pattern>%d{yyyy_MM_dd HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</Pattern>
   </encoder>
   <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
     <FileNamePattern>logs/linstor-${log.module}.%i.log.zip</FileNamePattern>
     <MinIndex>1</MinIndex>
     <MaxIndex>10</MaxIndex>
   </rollingPolicy>
   <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
     <MaxFileSize>2MB</MaxFileSize>
   </triggeringPolicy>
 </appender>
 <logger name="LINSTOR/Controller" level="INFO" additivity="false">
   <appender-ref ref="STDOUT" />
   <!-- <appender-ref ref="FILE" /> -->
 </logger>
 <logger name="LINSTOR/Satellite" level="INFO" additivity="false">
   <appender-ref ref="STDOUT" />
   <!-- <appender-ref ref="FILE" /> -->
 </logger>
 <root level="WARN">
   <appender-ref ref="STDOUT" />
   <!-- <appender-ref ref="FILE" /> -->
 </root>
</configuration>
----

See the https://logback.qos.ch/manual/index.html[Logback Manual] to find more details about `logback.xml`.


When none of the configuration methods above is used Linstor will default to `INFO` log level.

[[s-linstor-monitoring]]
=== Monitoring

Since LINSTOR 1.8.0, a https://prometheus.io/[Prometheus] `/metrics` HTTP path is provided with LINSTOR and JVM specific
exports.

The `/metrics` path also supports 3 GET arguments to reduce LINSTOR's reported data:

 * resource
 * storage_pools
 * error_reports

These are all default `true`, to disabled e.g. error-report data: `http://localhost:3370/metrics?error_reports=false`

==== Health check

The LINSTOR-Controller also provides a `/health` HTTP path that will simply return HTTP-Status 200 if the controller can access its
database and all services are up and running. Otherwise it will return HTTP error status code 500 `Internal Server Error`.


[[s-linstor-secure-connections]]
=== Secure Satellite connections

It is possible to have the LINSTOR use SSL secure TCP connection between controller and satellite connections.
Without going into further details on how java's SSL engine works we will give you
command line snippets using the `keytool` from java's runtime environment on how to configure
a 3 node setup using secure connections.
The node setup looks like this:

Node `alpha` is the just the controller.
Node `bravo` and node `charlie` are just satellites.


Here are the commands to generate such a keystore setup,
values should of course be edited for your environment.

---------
# create directories to hold the key files
mkdir -p /tmp/linstor-ssl
cd /tmp/linstor-ssl
mkdir alpha bravo charlie


# create private keys for all nodes
keytool -keyalg rsa -keysize 2048 -genkey -keystore alpha/keystore.jks\
 -storepass linstor -keypass linstor\
 -alias alpha\
 -dname "CN=Max Mustermann, OU=alpha, O=Example, L=Vienna, ST=Austria, C=AT"

keytool -keyalg rsa -keysize 2048 -genkey -keystore bravo/keystore.jks\
 -storepass linstor -keypass linstor\
 -alias bravo\
 -dname "CN=Max Mustermann, OU=bravo, O=Example, L=Vienna, ST=Austria, C=AT"

keytool -keyalg rsa -keysize 2048 -genkey -keystore charlie/keystore.jks\
 -storepass linstor -keypass linstor\
 -alias charlie\
 -dname "CN=Max Mustermann, OU=charlie, O=Example, L=Vienna, ST=Austria, C=AT"

# import truststore certificates for alpha (needs all satellite certificates)
keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore bravo/keystore.jks -destkeystore alpha/certificates.jks

keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore charlie/keystore.jks -destkeystore alpha/certificates.jks

# import controller certificate into satellite truststores
keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore alpha/keystore.jks -destkeystore bravo/certificates.jks

keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore alpha/keystore.jks -destkeystore charlie/certificates.jks

# now copy the keystore files to their host destinations
ssh root@alpha mkdir /etc/linstor/ssl
scp alpha/* root@alpha:/etc/linstor/ssl/
ssh root@bravo mkdir /etc/linstor/ssl
scp bravo/* root@bravo:/etc/linstor/ssl/
ssh root@charlie mkdir /etc/linstor/ssl
scp charlie/* root@charlie:/etc/linstor/ssl/

# generate the satellite ssl config entry
echo '[netcom]
  type="ssl"
  port=3367
  server_certificate="ssl/keystore.jks"
  trusted_certificates="ssl/certificates.jks"
  key_password="linstor"
  keystore_password="linstor"
  truststore_password="linstor"
  ssl_protocol="TLSv1.2"
' | ssh root@bravo "cat > /etc/linstor/linstor_satellite.toml"

echo '[netcom]
  type="ssl"
  port=3367
  server_certificate="ssl/keystore.jks"
  trusted_certificates="ssl/certificates.jks"
  key_password="linstor"
  keystore_password="linstor"
  truststore_password="linstor"
  ssl_protocol="TLSv1.2"
' | ssh root@charlie "cat > /etc/linstor/linstor_satellite.toml"
---------

Now just start controller and satellites and add the nodes with `--communication-type SSL`.

[[s-linstor-drbd-automatisms]]
=== Automatisms for DRBD-Resources

[[s-linstor-auto-quorum]]
==== AutoQuorum Policies

LINSTOR automatically configures quorum policies on resources *when
quorum is achievable*. This means, whenever you have at least two diskful and one
or more diskless resource assignments, or three or more diskful
resource assignments, LINSTOR will enable quorum policies for your
resources automatically.

Inversely, LINSTOR will automatically disable quorum policies whenever
there are less than the minimum required resource assignments to
achieve quorum.

This is controlled via the, `DrbdOptions/auto-quorum`, property which
can be applied to the _linstor-controller_, _resource-group_, and
_resource-definition_. Accepted values for the
`DrbdOptions/auto-quorum` property are `disabled`, `suspend-io`, and
`io-error`.

Setting the `DrbdOptions/auto-quorum` property to `disabled` will
allow you to manually, or more granularly, control the quorum policies
of your resources should you so desire.

TIP: The default policies for `DrbdOptions/auto-quorum` are `quorum
majority`, and `on-no-quorum io-error`. For more information on DRBD's
quorum features and their behavior, please refer to the
{url-drbd-ug}#s-feature-quorum[quorum section of the DRBD user's guide].

IMPORTANT: The `DrbdOptions/auto-quorum` policies will override any
manually configured properties if `DrbdOptions/auto-quorum` is not disabled.

For example, to manually set the quorum policies of a _resource-group_
named `my_ssd_group`, you would use the following commands:

----
# linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum majority
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum suspend-io
----

You may wish to disable DRBD's quorum features completely. To do that,
you would need to first disable `DrbdOptions/auto-quorum` on the
appropriate LINSTOR object, and then set the DRBD quorum features
accordingly. For example, use the following commands to disable quorum
entirely on the `my_ssd_group` _resource-group_:

----
# linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum off
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum
----

NOTE: Setting `DrbdOptions/Resource/on-no-quorum` to an empty value
in the commands above deletes the property from the object entirely.

[[s-linstor-auto-evict]]
==== Auto-Evict

If a satellite is offline for a prolonged period of time, LINSTOR can be configured to
declare that node as evicted. This triggers an automated reassignment of the affected
DRBD-resources to other nodes to ensure a minimum replica count is kept.

This feature uses the following properties to adapt the behaviour.

 * `DrbdOptions/AutoEvictMinReplicaCount` sets the number of replicas that should always
be present. You can set this property on the controller to change a global default, or on
a specific resource-definition or resource-group to change it only for that
resource-definition or resource-group. If this property is left empty, the place-count
set for the auto-placer of the corresponding resource-group will be used.

 * `DrbdOptions/AutoEvictAfterTime` describes how long a node can be offline in minutes
before the eviction is triggered. You can set this property on the controller to change a
global default, or on a single node to give it a different behavior. The default value
for this property is 60 minutes.

 * `DrbdOptions/AutoEvictMaxDisconnectedNodes` sets the percentage of nodes that can
be not reachable (for whatever reason) at the same time. If more than the given percent
of nodes are offline at the same time, the auto-evict will not be triggered for any node
, since in this case LINSTOR assumes connection problems from the controller. This property
can only be set for the controller, and only accepts a value between 0 and 100. The default
value is 34. If you wish to turn the auto-evict-feature off, simply set this property
to 0. If you want to always trigger the auto-evict, regardless of how many satellites
are unreachable, set it to 100.

 * `DrbdOptions/AutoEvictAllowEviction` is an additional property that can stop a node from
being evicted. This can be useful for various cases, for example if you need to shut down
a node for maintenance. You can set this property on the controller to change a global default,
or on a single node to give it a different behavior. It accepts true and false as values and
per default is set to true on the controller. You can use this property to turn the auto-evict
feature off by setting it to false on the controller, although this might not work
completely if you already set different values for individual nodes, since those values take precedence
over the global default.

After the linstor-controller loses the connection to a satellite, aside from trying to
reconnect, it starts a timer for that satellite. As soon as that timer exceeds
`DrbdOptions/AutoEvictAfterTime` and all of the DRBD-connections to the DRBD-resources
on that satellite are broken, the controller will check whether or not
`DrbdOptions/AutoEvictMaxDisconnectedNodes` has been met. If it hasn't, and
`DrbdOptions/AutoEvictAllowEviction` is true for the node in question, the satellite
will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether
the number of resources is still above `DrbdOptions/AutoEvictMinReplicaCount`. If it is,
the resource in question will be marked as DELETED. If it isn't, an auto-place with the settings
from the corresponding resource-group will be started. Should the auto-place fail, the
controller will try again later when changes that might allow a different result, such
as adding a new node, have happened. Resources where an auto-place is necessary will only be
marked as DELETED if the corresponding auto-place was successful.

The evicted satellite itself will not be able to reestablish connection with the controller.
Even if the node is up and running, a manual reconnect will fail. It is also not possible to
delete the satellite, even if it is working as it should be. The satellite can, however, be restored.
This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously
configured network interfaces, storage pools, properties and similar entities as well as
non-DRBD-related resources and resources that could not be autoplaced somewhere else will
still be on the satellite. To restore a satellite, use

------
# linstor node restore [nodename]
------

Should you wish to instead throw everything that once was on that node, including the node
itself, away, you need to use the `node lost` command instead.

[[s-linstor-qos]]
=== QoS Settings
==== Sysfs

LINSTOR is able to set the following Sysfs settings:

[cols="3,2"]
|===
| SysFs | Linstor property

| `/sys/fs/cgroup/blkio/blkio.throttle.read_bps_device` | `sys/fs/blkio_throttle_read`
| `/sys/fs/cgroup/blkio/blkio.throttle.write_bps_device` | `sys/fs/blkio_throttle_write`
| `/sys/fs/cgroup/blkio/blkio.throttle.read_iops_device` | `sys/fs/blkio_throttle_read_iops`
| `/sys/fs/cgroup/blkio/blkio.throttle.write_iops_device` | `sys/fs/blkio_throttle_write_iops`
|===

If a LINSTOR volume is composed of multiple "stacked" volume (for example DRBD with external
metadata will have 3 devices: backing (storage) device, metadata device and the resulting
DRBD device), setting a `sys/fs/\*` property for a Volume, only the bottom-most local
"data"-device will receive the corresponding `/sys/fs/cgroup/...` setting.
That means, in case of the example above only the backing device will receive the setting.
In case a resource-definition has an nvme-target as well as an nvme-initiator resource,
both bottom-most devices of each node will receive the setting. In case of the target the
bottom-most device will be the volume of LVM or ZFS, whereas in case of the initiator the
bottom-most device will be the connected nvme-device, regardless which other layers
are stacked on top of that.


[[s-linstor-getting-help]]
=== Getting help
==== From the command line
A quick way to list available commands on the command line is to type
`linstor`.

Further information on sub-commands (e.g., list-nodes) can be retrieved in
two ways:

----------------------------
# linstor node list -h
# linstor help node list
----------------------------

Using the 'help' sub-command is especially helpful when LINSTOR is executed
in interactive mode (`linstor interactive`).

One of the most helpful features of LINSTOR is its rich tab-completion,
which can be used to complete basically every object LINSTOR knows about
(e.g., node names, IP addresses, resource names, ...).
In the following examples, we show some possible completions, and their results:

----------------------------
# linstor node create alpha 1<tab> # completes the IP address if hostname can be resolved
# linstor resource create b<tab> c<tab> # linstor assign-resource backups charlie
----------------------------

If tab-completion does not work out of the box, please try to source the
appropriate file:

----------------------------
# source /etc/bash_completion.d/linstor # or
# source /usr/share/bash_completion/completions/linstor
----------------------------

For zsh shell users linstor-client can generate a zsh compilation file,
that has basic support for command and argument completion.

----------------------------
# linstor gen-zsh-completer > /usr/share/zsh/functions/Completion/Linux/_linstor
----------------------------

==== SOS-Report

If something goes wrong and you need help finding the cause of the issue, you can use

----------------------------
# linstor sos-report create
----------------------------

The command above will create a new sos-report in `/var/log/linstor/controller/` on the
controller node. Alternatively you can use

----------------------------
# linstor sos-report download
----------------------------

which will create a new sos-report and additionally downloads that report to the local machine
into your current working directory.

This sos-report contains logs and useful debug-information from several sources
(Linstor-logs, `dmesg`, versions of external tools used by Linstor, `ip a`, database dump
and many more).
These information are stored for each node in plaintext in the resulting `.tar.gz` file.


==== From the community
For help from the community please subscribe to our mailing list located here: https://lists.linbit.com/listinfo/drbd-user

==== GitHub
To file bug or feature request please check out our GitHub page https://github.com/linbit

==== Paid support and development
Alternatively, if you wish to purchase remote installation services, 24/7 support, access to certified repositories, or feature development please contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | sales@linbit.com
