[[ch-linstor-gateway]]
== Exporting Highly Available Storage using LINSTOR Gateway

*LINSTOR Gateway* manages highly available iSCSI targets and NFS exports by leveraging on LINSTOR and
Pacemaker. Setting up LINSTOR - including storage pools and resource groups - as well as Corosync and
Pacemaker's properties are a prerequisite to use this tool.

[[s-linstor-gateway-requirements]]
=== Requirements

==== LINSTOR

A LINSTOR cluster is required to operate LINSTOR Gateway. It is *highly recommended* to run the LINSTOR
controller as a Pacemaker resource. This needs to be configured manually. Such a resource could look like the
following: (optional)

-----------------
primitive p_linstor-controller systemd:linstor-controller \
        op start interval=0 timeout=100s \
        op stop interval=0 timeout=100s \
        op monitor interval=30s timeout=100s
-----------------

For both iSCSI and NFS, `storage pool` , `resource group` and a `volume group` for LINSTOR Gateway needs to be
present. Lets do it;

Creating the storage pool in 3 nodes using the phsical device `/dev/sdb`;

-----------------
linstor physical-storage create-device-pool --pool-name lvpool LVM LINSTOR1 /dev/sdb --storage-pool lvmpool
linstor physical-storage create-device-pool --pool-name lvpool LVM LINSTOR2 /dev/sdb --storage-pool lvmpool
linstor physical-storage create-device-pool --pool-name lvpool LVM LINSTOR3 /dev/sdb --storage-pool lvmpool
-----------------

We also need resource groups and volume groups;

-----------------
linstor rg c iSCSI_group --storage-pool lvmpool --place-count 2
linstor rg c nfs_group --storage-pool lvmpool --place-count 3
-----------------

-----------------
linstor vg c iSCSI_group
linstor vg c nfs_group
-----------------

For a more detailed explanation of the storage pool, resource group and volume group creation, check the
https://linbit.com/drbd-user-guide/LINSTOR-guide-1_0-en/#s-storage_pools[LINSTOR user guide]

==== Pacemaker

A working Corosync/Pacemaker cluster is expected on the machine where LINSTOR Gateway is running.
The https://github.com/LINBIT/drbd-utils/blob/master/scripts/drbd-attr[drbd-attr] resource agent is required
to run LINSTOR Gateway. This is included in LINBIT's `drbd-utils` package for Ubuntu based distributions, or
the drbd-pacemaker package on RHEL/CentOS.  LINSTOR Gateway sets up all required Pacemaker resource and
constraints by itself, except for the LINSTOR controller resource.

==== iSCSI & NFS

LINSTOR Gateway uses Pacemaker's `ocf::heartbeat:iSCSITarget` resource agent for its iSCSI integration, which
requires an iSCSI implementation to be installed. Using `targetcli` is recommended.

For iSCSI, please install targetcli.

-----------------
yum install targetcli
-----------------

For nfs, nfs-server needs to be enabled and ready;

-----------------
systemctl enable --now nfs-server
-----------------

[[s-linstor-gateway-preparation]]
=== Preparation
First, let's check that all the components are available. This guide assumes you already installed and
configured a LINSTOR cluster. Volume Group, Storage Pool and Resource Group should be defined before using
linstor-iscsi or linstor-nfs.

Tools need to be present in server;

* linstor-client (managing the LINSTOR cluster)
* drbd-attr resource agent (part of drbd-utils in Debian/Ubuntu, and part of drbd-pacemaker for other
  Linux distrubutions)
* targetcli (for iSCSI)
* nfs-utils, nfs-server
* pcs or crmsh for pacemaker client. (checking the status of the iSCSI or nfs targets)

[[s-linstor-gateway-checking]]
=== Checking the Cluster

Check the LINSTOR cluster status with;

-----------------
[root@LINSTOR1 ~]# linstor n l
╭────────────────────────────────────────────────────────────╮
┊ Node     ┊ NodeType  ┊ Addresses                  ┊ State  ┊
╞════════════════════════════════════════════════════════════╡
┊ LINSTOR1 ┊ COMBINED  ┊ 172.16.16.111:3366 (PLAIN) ┊ Online ┊
┊ LINSTOR2 ┊ SATELLITE ┊ 172.16.16.112:3366 (PLAIN) ┊ Online ┊
┊ LINSTOR3 ┊ SATELLITE ┊ 172.16.16.113:3366 (PLAIN) ┊ Online ┊
╰────────────────────────────────────────────────────────────╯
-----------------
Check the LINSTOR storage pool list with;

-----------------
root@LINSTOR1 ~]# linstor sp l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node     ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ LINSTOR1 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ LINSTOR2 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ LINSTOR3 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊
┊ lvmpool              ┊ LINSTOR1 ┊ LVM      ┊ lvpool   ┊    10.00 GiB ┊     10.00 GiB ┊ False        ┊ Ok    ┊
┊ lvmpool              ┊ LINSTOR2 ┊ LVM      ┊ lvpool   ┊    10.00 GiB ┊     10.00 GiB ┊ False        ┊ Ok    ┊
┊ lvmpool              ┊ LINSTOR3 ┊ LVM      ┊ lvpool   ┊    10.00 GiB ┊     10.00 GiB ┊ False        ┊ Ok    ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
-----------------

LINSTOR resource group list (please do not forget to create volume group for the resource group with:
`linstor vg c iscsi_group`)

-----------------
[root@LINSTOR1 ~]# linstor rg l
╭────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter            ┊ VlmNrs ┊ Description ┊
╞════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp    ┊ PlaceCount: 2           ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ iscsi_group   ┊ PlaceCount: 2           ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): lvmpool ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ nfs_group     ┊ PlaceCount: 3           ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): lvmpool ┊        ┊             ┊
╰────────────────────────────────────────────────────────────────╯

-----------------

Check and disable stonith. Stonith is a technique for fencing node in clusters. We have 3 nodes here so quorum
will be used instead of fencing.

-----------------
pcs property set stonith-enabled=false
pcs property show
-----------------

Check the pacemaker cluster health

-----------------
[root@LINSTOR1 ~]# pcs status
Cluster name: LINSTOR
Cluster Summary:
  * Stack: corosync
  * Current DC: LINSTOR1 (version 2.0.5.linbit-1.0.el7-ba59be712) - partition with quorum
  * Last updated: Wed Mar 24 21:24:10 2021
  * Last change:  Wed Mar 24 21:24:01 2021 by root via cibadmin on LINSTOR1
  * 3 nodes configured
  * 0 resource instances configured

Node List:
  * Online: [ LINSTOR1 LINSTOR2 LINSTOR3 ]

Full List of Resources:
  * No resources

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
-----------------

[[s-linstor-gateway-setup-iscsi]]
=== Setting up iSCSI target


Now, everything looks good, let's start creating our first iSCSI lun. linstor-iscsi tool will be used for all
iSCSI related actions. Please check "linstor-iscsi help" for detailed usage. At first it creates a new
resource within the LINSTOR system under the specified name and using the specified resource group. After that
it creates resource primitives in the Pacemaker cluster including all necessary order and location
constraints. The Pacemaker primitives are prefixed with `p_`, contain the resource name and a resource type
postfix.

------------------------
linstor-iscsi create --iqn=iqn.2021-04.com.linbit:lun4 --ip=172.16.16.101/24 --username=foo --lun=4 --password=bar --resource-group=iSCSI_group --size=1G
------------------------

This command will create a 1G iSCSI disk with the provided username and password in the resource group defined
`iSCSI_group` DRBD and pacemaker resources will be automatically created by linstor-iscsi. You can check the
pacemaker resources with "pcs status" command.

------------------------
[root@LINSTOR1 ~]# linstor-iscsi list
+-----------------------------+-----+---------------+-----------+--------------+---------+
|             IQN             | LUN | Pacemaker LUN | Pacemaker | Pacemaker IP | LINSTOR |
+-----------------------------+-----+---------------+-----------+--------------+---------+
| iqn.2020-06.com.linbit:lun4 |   4 |       ✓       |     ✓     |      ✓       |    ✓    |
+-----------------------------+-----+---------------+-----------+--------------+---------+
------------------------

[[s-linstor-gateway-delete-iscsi]]
=== Deleting iSCSI target

The following command will delete the iSCSI target from pacemaker as well as the LINSTOR cluster;

------------------------
linstor-iscsi delete -i iqn.2021-04.com.linbit:lun4 -l 4
------------------------

[[s-linstor-gateway-setup-nfs]]
=== Setting up NFS export

Before creating the nfs exports you need to tell LINSTOR that filesystem will be used for NFS exports will be
EXT4. in order to do that, we'll apply a property to the resource group of NFS resources simply by typing;

------------------------
linstor rg set-property nfs_group FileSystem/Type ext4
------------------------

The following command will create a NFS export in the cluster. At first it creates a new resource within the
LINSTOR system under the specified name and using the specified resource group. After that it creates resource
primitives in the Pacemaker cluster including all necessary order and location constraints. The Pacemaker
primites are prefixed with p_, contain the resource name and a resource type postfix.

------------------------
linstor-nfs create --resource=nfstest --service-ip=172.16.16.102/32 --allowed-ips=172.16.16.0/24 --resource-group=nfs_group --size=1G
------------------------

You can simply list the nfs exports with the command below;

------------------------
[root@LINSTOR1 ~]# LINSTOR-nfs list
+---------------+------------------+-----------------------+------------+------------+
| Resource name | LINSTOR resource | Filesystem mountpoint | NFS export | Service IP |
+---------------+------------------+-----------------------+------------+------------+
| nfstest       |        ✓         |           ✓           |     ✓      |     ✓      |
+---------------+------------------+-----------------------+------------+------------+
------------------------

[[s-linstor-gateway-delete-nfs]]
=== Deleting NFS Export

The following command will delete the nfs export from pacemaker as well as the LINSTOR cluster;

------------------------
[root@LINSTOR1 ~]# linstor-nfs delete -r nfstest
------------------------
