[[ch-nomad-linstor]]
== LINSTOR Volumes in Nomad

indexterm:[Nomad]

:consul-connect: https://www.nomadproject.io/docs/integrations/consul-integration#service-discovery

This chapter describes using LINSTOR and DRBD to provision volumes in Nomad.

[[s-nomad-linstor-overview]]
=== Nomad Overview

https://www.nomadproject.io/[Nomad] is a simple and flexible workload orchestrator to deploy and manage containers
and non-containerized applications across on-prem and clouds.

Nomad supports provisioning storage volumes via https://www.nomadproject.io/docs/internals/plugins/csi[plugins]
conforming to the https://github.com/container-storage-interface/spec[Container Storage Interface (CSI)].

LINBIT distributes a CSI plugin in the form of container images from http://drbd.io[drbd.io]. The plugin can be
configured to work with a LINSTOR cluster that is deployed along or inside a Nomad cluster.

[[s-nomad-linstor-deployment]]
=== Deploying LINSTOR on Nomad

This section describes how you can deploy and configure a new LINSTOR cluster in Nomad.

NOTE: If you want to install LINSTOR directly on your nodes, check out the
<<s-installation, guide on installing LINSTOR>>. You can skip this section and jump directly to
<<s-nomad-linstor-csi-deployment, deploying the CSI driver>>.

[[s-nomad-prepare]]
==== Preparing Nomad

In order to run LINSTOR, every Nomad agent needs to be configured to:

- Support the https://www.nomadproject.io/docs/drivers/docker[docker driver] and allow executing privileged containers
+
To allow running privileged containers, add the following snippet to your Nomad agent configuration and restart Nomad
+
./etc/nomad.d/docker-privileged.hcl
[source,hcl]
----
plugin "docker" {
  config {
    allow_privileged = true
  }
}
----

- Support for container networking. If you don't have the https://www.cni.dev/[Container Network Interface] plugins
installed, you will only be able to use `mode = "host"` in your job networks. For most production setups, we recommend
installing the default plugins:
+
Head to the https://github.com/containernetworking/plugins/releases/[plugin release page], select the release archive
appropriate for your distribution and unpack them in `/opt/cni/bin`. You might need to create the directory before
unpacking.

- Provide a host volume, allowing a container access to the hosts `/dev` directory
+
To create a host volume, add the following snippet to your Nomad agent configuration and restart Nomad
+
./etc/nomad.d/host-volume-dev.hcl
[source,hcl]
----
client {
  host_volume "dev" {
    path = "/dev"
  }
}
----

==== Create a LINSTOR Controller Job

The LINSTOR Controller is deployed as a service with no replicas. At any point in time, there can only be one LINSTOR
Controller running in a cluster. It is possible to restart the controller on a new node, provided it still has
access to the database. See <<s-linstor_ha>> for more information.

The following example will create a Nomad job starting a single LINSTOR Controller in datacenter `dc1` and connect
to an external database.

.linstor-controller.hcl
[source,hcl]
----
job "linstor-controller" {
  datacenters = ["dc1"] <1>
  type = "service"

  group "linstor-controller" {
    network {
      mode = "bridge"
      # port "linstor-api" { <2>
      #   static = 3370
      #   to = 3370
      # }
    }

    service { <3>
      name = "linstor-api"
      port = "3370"

      connect {
        sidecar_service {}
      }

      check {
        expose = true
        type = "http"
        name = "api-health"
        path = "/health"
        interval = "30s"
        timeout = "5s"
      }
    }

    task "linstor-controller" {
      driver = "docker"
      config {
        image = "drbd.io/linstor-controller:v1.13.0" <4>

        auth { <5>
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }

        mount {
          type = "bind"
          source = "local"
          target = "/etc/linstor"
        }
      }

      # template { <6>
      #  destination = "local/linstor.toml"
      #  data = <<EOH
      #    [db]
      #    user = "example"
      #    password = "example"
      #    connection_url = "jdbc:postgresql://postgres.internal.example.com/linstor"
      #  EOH
      # }

      resources {
        cpu    = 500 # 500 MHz
        memory = 700 # 700MB
      }
    }
  }
}
----

<1> Replace `dc1` with your own datacenter name

<2> This exposes the LINSTOR API on the host on port `3370`.
+
NOTE: Uncomment this section if your cluster is not configured with {consul-connect}[Consul Connect].

<3> The `service` block is used to expose the LINSTOR API to other jobs via the service mesh.
+
NOTE: If your cluster is not configured for {consul-connect}[Consul Connect] you can remove this section.

<4> This sets the LINSTOR Controller image to run. The latest images are available from http://drbd.io[drbd.io].
+
IMPORTANT: The use of the `:latest` tag is strongly discouraged, as it can quickly lead to version mismatches and
unintended upgrades.

<5> Sets the authentication to use when pulling the image. If pulling from `drbd.io`, you need to use your LINBIT
customer login here. Read more about pulling from a private repo
https://www.nomadproject.io/docs/drivers/docker#authentication[here].

<6> This template can be used to set arbitrary configuration options for LINSTOR. This example configures an external
database for LINSTOR. You can find a more detailed explanation of LINSTORs database options
<<s-linstor-external-database,here>> and more on Nomad templates
https://www.nomadproject.io/docs/job-specification/template#template-examples[here].

Apply the job by running:

[source,shell]
----
$ nomad job run linstor-controller.hcl
==> Monitoring evaluation "7d8911a7"
    Evaluation triggered by job "linstor-controller"
==> Monitoring evaluation "7d8911a7"
    Evaluation within deployment: "436f4b2d"
    Allocation "0b564c73" created: node "07754224", group "controller"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "7d8911a7" finished with status "complete"
----

===== Using a host volume for Linstors database

If you want to try LINSTOR without setting up an external database, you can make use of Linstors built-in filesystem
based database. To make the database persistent, you need to ensure it is placed on a host volume.

IMPORTANT: Using a host volume means that only a single node is able to run the LINSTOR Controller. If the node is
unavailable, the LINSTOR Cluster will also be unavailable. For alternatives, use an external (highly available) database
or deploy the <<s-linstor_ha,LINSTOR cluster directly on the hosts>>.

To create a host volume for the LINSTOR database, first create the directory on the host with the expected permissions

[source,shell]
----
$ mkdir -p /var/lib/linstor
$ chown -R 1000:0 /var/lib/linstor
----

Then add the following snippet to your Nomad agent configuration and restart Nomad

./etc/nomad.d/host-volume-linstor-db.hcl
[source,hcl]
----
client {
  host_volume "linstor-db" {
    path = "/var/lib/linstor"
  }
}
----

Then, add the following snippets to the `linstor-controller.hcl` example from above and adapt the `connection_url`
option from the configuration template.

.`job > group`
[source,hcl]
----
volume "linstor-db" {
  type = "host"
  source = "linstor-db"
}
----

.`job > group > task`
[source,hcl]
----
volume_mount {
  volume = "linstor-db"
  destination = "/var/lib/linstor"
}

template {
  destination = "local/linstor.toml"
  data = <<EOH
    [db]
    user = "linstor"
    password = "linstor"
    connection_url = "jdbc:h2:/var/lib/linstor/linstordb"
  EOH
}
----

==== Create a LINSTOR Satellite job

The LINSTOR Satellites are deployed as a system job in Nomad, running in a privileged container. In addition to the
satellites, the job will also load the DRBD module along with other kernel modules used by LINSTOR.

The following example will create a Nomad job starting a LINSTOR satellite on every node in datacenter `dc1`.

.linstor-satellite.hcl
[source,hcl]
----
job "linstor-satellite" {
  datacenters = ["dc1"] <1>
  type = "system"

  group "satellite" {
    network {
      mode = "host"
    }

    volume "dev" { <2>
      type = "host"
      source = "dev"
    }

    task "linstor-satellite" {
      driver = "docker"

      config {
        image = "drbd.io/linstor-satellite:v1.13.0" <3>

        auth { <4>
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }

        privileged = true <5>
        network_mode = "host" <6>
      }

      volume_mount { <2>
        volume = "dev"
        destination = "/dev"
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 500 # 500MB
      }
    }

    task "drbd-loader" {
      driver = "docker"
      lifecycle {
        hook = "prestart" <7>
      }

      config {
        image = "drbd.io/drbd9-rhel8:v9.0.29" <8>

        privileged = true <5>
        auth { <4>
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }
      }

      env {
        LB_HOW = "shipped_modules" <9>
      }

      volume_mount { <10>
        volume = "kernel-src"
        destination = "/usr/src"
      }
      volume_mount { <10>
        volume = "modules"
        destination = "/lib/modules"
      }
    }

    volume "modules" { <10>
      type = "host"
      source = "modules"
      read_only = true
    }

    volume "kernel-src" { <10>
      type = "host"
      source = "kernel-src"
      read_only = true
    }
  }
}
----

<1> Replace `dc1` with your own datacenter name

<2> The `dev` host volume is the volume created in <<s-nomad-prepare>>, which allows the satellite to manage the hosts
block devices.

<3> This sets the LINSTOR Satellite image to run. The latest images are available from http://drbd.io[drbd.io]. The
satellite image version has to match the version of the controller image.
+
IMPORTANT: The use of the `:latest` tag is strongly discouraged, as it can quickly lead to version mismatches and
unintended upgrades.

<4> Sets the authentication to use when pulling the image. If pulling from `drbd.io`, you need to use your LINBIT
customer login here. Read more about pulling from a private repo
https://www.nomadproject.io/docs/drivers/docker#authentication[here].

<5> In order to configure storage devices, DRBD and load kernel modules, the containers need to be running in privileged
mode.

<6> The satellite needs to communicate with DRBD, which requires access to the netlink interface running in the hosts
network.

<7> The `drbd-loader` task will be executed once at the start of the satellite and load DRBD and other useful kernel
modules.

<8> The `drbd-loader` is specific to the distribution you are using. Available options are:
* `drbd.io/drbd9-bionic` for Ubuntu 18.04 (Bionic Beaver)
* `drbd.io/drbd9-focal` for Ubuntu 20.04 (Focal Fossa)
* `drbd.io/drbd9-rhel8` for RHEL 8
* `drbd.io/drbd9-rhel7` for RHEL 7

<9> The `drbd-loader` container can be configured via environment variables. `LB_HOW` tells the container how to insert
the DRBD kernel module. Available options are:
`shipped_modules`:: uses the prepackaged RPMs or DEBs delivered with the container.
`compile`:: Compile DRBD from source. Requires access to the kernel headers (see below).
`deps_only`:: Only try to load existing modules used by the LINSTOR satellite (for example `dm_thin_pool` and `dm_cache`).

<10> In order for the `drbd-loader` container to build DRBD or load existing modules, it needs access to a hosts
`/usr/src` and `/lib/modules` respectively.
+
This requires setting up additional host volumes on every node. The following snippet needs to be added to every Nomad
agent confiration, then Nomad needs to be restarted.
+
./etc/nomad.d/drbd-loader-volumes.hcl
[source,hcl]
----
client {
  host_volume "modules" {
    path = "/lib/modules"
    read_only = true
  }
  host_volume "kernel-src" {
    path = "/usr/src"
    read_only = true
  }
}
----

Apply the job by running:

[source,shell]
----
$ nomad job run linstor-satellite.hcl
==> Monitoring evaluation "0c07469d"
    Evaluation triggered by job "linstor-satellite"
==> Monitoring evaluation "0c07469d"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "0c07469d" finished with status "complete"
----

==== Configuring LINSTOR in Nomad

Once the `linstor-controller` and `linstor-satellite` jobs are running, you can start configuring the cluster using
the `linstor` command line tool.

This can done:

* directly by `nomad exec`-ing into the `linstor-controller` container
* using the `drbd.io/linstor-client` container on the host running the `linstor-controller`
+
----
docker run -it --rm --net=host drbd.io/linstor-client node create
----
* by installing the `linstor-client` package on the host running the `linstor-controller`.

In all cases, you need to <<s-adding_nodes_to_your_cluster,add the satellites to your cluster>> and
<<s-storage_pools,create some storage pools>>. For example, to add the node `nomad-01.example.com` and configure
a LVM Thin storage pool, you would run:

[source,shell]
----
$ linstor node create nomad-01.example.com
$ linstor storage-pool create lvmthin nomad-01.example.com thinpool linstor_vg/thinpool
----

IMPORTANT: The CSI driver requires your satellites to be named after their hostname. To be precise, the satellite name
needs to match Nomads `attr.unique.hostname` attribute on the node.

[[s-nomad-linstor-csi-deployment]]
=== Deploying the LINSTOR CSI Driver on Nomad

The CSI driver is deployed as a system job, meaning it runs on every node in the cluster.

The following example will create a Nomad job starting a LINSTOR CSI Driver on every node in datacenter `dc1`.

.linstor-csi.hcl
[source,hcl]
----
job "linstor-csi" {
  datacenters = ["dc1"] <1>
  type = "system"

  group "csi" {
    network {
      mode = "bridge"
    }

    service {
      connect {
        sidecar_service { <2>
          proxy {
            upstreams {
              destination_name = "linstor-api"
              local_bind_port  = 8080
            }
          }
        }
      }
    }

    task "csi-plugin" {
      driver = "docker"
      config {
        image = "drbd.io/linstor-csi:v0.13.1" <3>

        auth { <4>
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }

        args = [
          "--csi-endpoint=unix://csi/csi.sock",
          "--node=${attr.unique.hostname}", <5>
          "--linstor-endpoint=http://${NOMAD_UPSTREAM_ADDR_linstor_api}", <6>
          "--log-level=info"
        ]

        privileged = true <7>
      }

      csi_plugin { <8>
        id = "linstor.csi.linbit.com"
        type = "monolith"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 100 # 100 MHz
        memory = 200 # 200MB
      }
    }
  }
}
----

<1> Replace `dc1` with your own datacenter name

<2> The `sidecar_service` stanza enables use of the service mesh generated by using {consul-connect}[Consul Connect].
If you have not configured this feature in Nomad, or you are using an external LINSTOR Controller,
you can skip this configuration.

<3> This sets the LINSTOR CSI Driver image to run. The latest images are available from http://drbd.io[drbd.io].
+
IMPORTANT: The use of the `:latest` tag is strongly discouraged, as it can quickly lead to version mismatches and
unintended upgrades.

<4> Sets the authentication to use when pulling the image. If pulling from `drbd.io`, you need to use your LINBIT
customer login here. Read more about pulling from a private repo
https://www.nomadproject.io/docs/drivers/docker#authentication[here].

<5> This argument sets the node name used by the CSI driver to identify itself in the LINSTOR API. By default, this
is set to the nodes hostname.

<6> This argument sets the LINSTOR API endpoint. If you are not using the consul service mesh (see Nr. 2 above), this
needs to be set to the Controllers API endpoint. The endpoint needs to be reachable from every node this is deployed on.

<7> The CSI driver needs to execute mount commands, requiring privileged containers.

<8> The `csi_plugin` stanza informs Nomad that this task is a CSI plugin. The Nomad agent will forward requests for
volumes to one of the jobs containers.

Apply the job by running:

[source,shell]
----
$ nomad job run linstor-csi.hcl
==> Monitoring evaluation "0119f19c"
    Evaluation triggered by job "linstor-csi"
==> Monitoring evaluation "0119f19c"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "0119f19c" finished with status "complete"
----

=== Using LINSTOR volumes in Nomad

Volumes in Nomad are created using a https://www.nomadproject.io/docs/commands/volume/create#volume-specification[volume-specification].

As an example, the following specification requests a 1GiB volume with 2 replicas from the LINSTOR storage pool `thinpool`.

.vol1.hcl
[source,hcl]
----
id = "vol1" <1>
name = "vol1" <2>

type = "csi"
plugin_id = "linstor.csi.linbit.com"

capacity_min = "1GiB"
capacity_max = "1GiB"

capability {
  access_mode = "single-node-writer" <3>
  attachment_mode = "file-system" <4>
}

mount_options {
  fs_type = "ext4" <5>
}

parameters { <6>
  "resourceGroup" = "default-resource",
  "storagePool" = "thinpool",
  "autoPlace" = "2"
}
----

<1> The `id` is used to reference this volume in Nomad. Used in the `volume.source` field of a job specification.

<2> The `name` is used when creating the volume in the backend (i.e. LINSTOR). Ideally this matches the `id` and is a
valid LINSTOR resource name. If the name would not be valid, LINSTOR CSI will generate a random compatible name.

<3> What kind of access the volume should support. LINSTOR CSI supports:
`single-node-reader-only`::Allow read only access on one node at a time
`single-node-writer`:: Allow read and write access on one node at a time
`multi-node-reader-only`:: Allow read only access from multiple nodes.

<4> Can be `file-system` or `block-device`.

<5> Specify the file system to use. LINSTOR CSI supports `ext4` and `xfs`.

<6> Additional parameters to pass to LINSTOR CSI. The example above requests the resource be part of the
`default-resource` <<s-linstor-resource-groups,resource group>> and should deploy 2 replicas.
+
For a complete list of available parameters, you can check out the
<<s-kubernetes-sc-parameters,guide on Kubernetes storage classes>>. Kubernetes, like Nomad, makes use of the CSI plugin.

To create the volume, run the following command:

[source,shell]
----
$ nomad volume create vol1.hcl
Created external volume vol1 with ID vol1
$ nomad volume status
Container Storage Interface
ID    Name  Plugin ID               Schedulable  Access Mode
vol1  vol1  linstor.csi.linbit.com  true         <none>
$ linstor resource list
╭──────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node                 ┊ Port ┊ Usage  ┊ Conns ┊    State ┊ CreatedOn           ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════╡
┊ vol1         ┊ nomad-01.example.com ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2021-06-15 14:56:32 ┊
┊ vol1         ┊ nomad-02.example.com ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2021-06-15 14:56:32 ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
----

==== Using volumes in jobs

To use the volume in a job, add the `volume` and `volume_mount` stanzas to the job specification:

[source,hcl]
----
job "example" {
  ...

  group "example" {
    volume "example-vol" {
      type = "csi"
      source = "vol1"
      attachment_mode = "file-system"
      access_mode = "single-node-writer"
    }

    task "mount-example" {
      volume_mount {
        volume = "example-vol"
        destination = "/data"
      }

      ...
    }
  }
}
----

==== Creating snapshots of volumes

LINSTOR can create snapshots of existing volumes, provided the underlying storage pool driver supports snapshots.

The following command creates a snapshot named `snap1` of the volume `vol1`.

[source,shell]
----
$ nomad volume snapshot create vol1 snap1
Snapshot ID  Volume ID  Size     Create Time  Ready?
snap1        vol1       1.0 GiB  None         true
$ linstor s l
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ SnapshotName ┊ NodeNames                                  ┊ Volumes  ┊ CreatedOn           ┊ State      ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ vol1         ┊ snap1        ┊ nomad-01.example.com, nomad-02.example.com ┊ 0: 1 GiB ┊ 2021-06-15 15:04:10 ┊ Successful ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
----

You can use a snapshot to pre-populate an existing volume with data from the snapshot

[source,shell]
----
$ cat vol2.hcl
id = "vol2"
name = "vol2"
snapshot_id = "snap1"

type = "csi"
plugin_id = "linstor.csi.linbit.com"
...

$ nomad volume create vol2.hcl
Created external volume vol2 with ID vol2
----
