[[ch-gfs]]
== Using GFS with DRBD

indexterm:[GFS]indexterm:[Global File System]This chapter outlines the
steps necessary to set up a DRBD resource as a block device holding a
shared Global File System (GFS). It covers both GFS and GFS2.

In order to use GFS on top of DRBD, you must configure DRBD in
indexterm:[dual-primary mode]<<s-dual-primary-mode,dual-primary
mode>>.

[IMPORTANT]
===============================
All cluster file systems _require_ fencing - not only via the DRBD
resource, but STONITH! A faulty member _must_ be killed.

You'll want these settings:

	net {
		fencing resource-and-stonith;
	}
	handlers {
		# Make sure the other node is confirmed
		# dead after this!
		outdate-peer "/sbin/kill-other-node.sh";
	}

There must be _no_ volatile caches!
Please see https://fedorahosted.org/cluster/wiki/DRBD_Cookbook for some more information.
===============================


[[s-gfs-primer]]
=== GFS primer

The Red Hat Global File System (GFS) is Red Hat's implementation of a
concurrent-access shared storage file system. As any such filesystem,
GFS allows multiple nodes to access the same storage device, in
read/write fashion, simultaneously without risking data corruption. It
does so by using a Distributed Lock Manager (DLM) which manages
concurrent access from cluster members.

GFS was designed, from the outset, for use with conventional shared
storage devices. Regardless, it is perfectly possible to use DRBD, in
dual-primary mode, as a replicated storage device for
GFS. Applications may benefit from reduced read/write latency due to
the fact that DRBD normally reads from and writes to local storage, as
opposed to the SAN devices GFS is normally configured to run
from. Also, of course, DRBD adds an additional physical copy to every
GFS filesystem, thus adding redundancy to the concept.

GFS makes use of a cluster-aware variant of LVM, indexterm:[LVM]termed
Cluster Logical Volume Manager or indexterm:[CLVM]CLVM. As such, some parallelism
exists between using DRBD as the data storage for GFS, and using
<<s-lvm-drbd-as-pv,DRBD as a Physical Volume for conventional LVM>>.

GFS file systems are usually tightly integrated with Red Hat's own
cluster management framework, the indexterm:[Red Hat Cluster
Suite]<<ch-rhcs,Red Hat Cluster>>. This chapter explains
the use of DRBD in conjunction with GFS in the Red Hat Cluster context.

GFS, CLVM, and Red Hat Cluster are available in Red Hat
Enterprise Linux (RHEL) and distributions derived from it, such as
indexterm:[CentOS]CentOS. Packages built from the same sources are
also available in indexterm:[Debian GNU/Linux]Debian GNU/Linux. This
chapter assumes running GFS on a Red Hat Enterprise Linux system.

[[s-gfs-create-resource]]
=== Creating a DRBD resource suitable for GFS

Since GFS is a shared cluster file system expecting concurrent
read/write storage access from all cluster nodes, any DRBD resource to
be used for storing a GFS filesystem must be configured in
<<s-dual-primary-mode,dual-primary mode>>. Also, it is recommended to
use some of DRBD's
<<s-automatic-split-brain-recovery-configuration,features for
automatic recovery from split brain>>. To
do all this, include the following lines in the resource
configuration: indexterm:[drbd.conf]

[source,drbd]
----------------------------
resource <resource> {
  net {
    allow-two-primaries yes;
    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;
    ...
  }
  ...
}
----------------------------

[WARNING]
===============================
By configuring auto-recovery policies, you are configuring effectively configuring automatic data-loss! Be sure you understand the implications.
===============================


Once you have added these options to <<ch-configure,your
freshly-configured resource>>, you may <<s-first-time-up,initialize
your resource as you normally would>>. Since the
indexterm:[drbd.conf]`allow-two-primaries` option is set to `yes` for
this resource, you will be able to <<s-switch-resource-roles,promote
the resource>>to the primary role on two nodes.

[[s-gfs-configure-lvm]]
=== Configuring LVM to recognize the DRBD resource

GFS uses CLVM, the cluster-aware version of LVM, to manage block
devices to be used by GFS. In order to use CLVM with DRBD, ensure that
your LVM configuration

* uses clustered locking. To do this, set the following option in
  `/etc/lvm/lvm.conf`:
+
[source,drbd]
----------------------------
locking_type = 3
----------------------------

* scans your DRBD devices to recognize DRBD-based Physical Volumes
  (PVs). This applies as to conventional (non-clustered) LVM; see
  <<s-lvm-drbd-as-pv>> for details.

[[s-gfs-enable]]
=== Configuring your cluster to support GFS

After you have created your new DRBD resource and
<<s-rhcs-config,completed your initial cluster configuration>>, you
must enable and start the following system services on both nodes of
your GFS cluster:

* `cman` (this also starts `ccsd` and `fenced`),

* `clvmd`.



[[s-gfs-create]]
=== Creating a GFS filesystem

In order to create a GFS filesystem on your dual-primary DRBD
resource, you must first initialize it as a <<s-lvm-primer,Logical
Volume for LVM>>.

Contrary to conventional, non-cluster-aware LVM configurations, the
following steps must be completed on only one node due to the
cluster-aware nature of CLVM: indexterm:[LVM]indexterm:[pvcreate (LVM
command)]indexterm:[LVM]indexterm:[vgcreate (LVM
command)]indexterm:[LVM]indexterm:[lvcreate (LVM command)]

----------------------------
# pvcreate /dev/drbd/by-res/<resource>/0
Physical volume "/dev/drbd<num>" successfully created
# vgcreate <vg-name> /dev/drbd/by-res/<resource>/0
Volume group "<vg-name>" successfully created
# lvcreate --size <size> --name <lv-name> <vg-name>
Logical volume "<lv-name>" created
----------------------------

NOTE: This example assumes a single-volume resource.

CLVM will immediately notify the peer node of these changes;
indexterm:[LVM]indexterm:[lvdisplay (LVM
command)]indexterm:[LVM]indexterm:[lvs (LVM command)] issuing `lvs`
(or `lvdisplay`) on the peer node will list the newly created logical
volume.

indexterm:[GFS]Now, you may proceed by creating the actual filesystem:
----------------------------
# mkfs -t gfs -p lock_dlm -j 2 /dev/<vg-name>/<lv-name>
----------------------------

Or, for a GFS2 filesystem:

----------------------------
# mkfs -t gfs2 -p lock_dlm -j 2 -t <cluster>:<name>
	/dev/<vg-name>/<lv-name>
----------------------------

The `-j` option in this command refers to the number of journals to
keep for GFS. This must be identical to the number of nodes with concurrent _Primary_ role in the GFS
cluster; since DRBD does not support more than two _Primary_ nodes until DRBD 9.1, the value to
set here is always 2.

The `-t` option, applicable only for GFS2 filesystems, defines the lock
table name. This follows the format _<cluster>:<name>_, where _<cluster>_
must match your cluster name as defined in
`/etc/cluster/cluster.conf`. Thus, only members of that cluster will
be permitted to use the filesystem. By contrast, _<name>_ is an
arbitrary file system name unique in the cluster.

[[s-gfs-use]]
=== Using your GFS filesystem

After you have created your filesystem, you may add it to
`/etc/fstab`:

[source,fstab]
----------------------------
/dev/<vg-name>/<lv-name> <mountpoint> gfs defaults 0 0
----------------------------

For a GFS2 filesystem, simply change the filesystem type:

[source,fstab]
----------------------------
/dev/<vg-name>/<lv-name> <mountpoint> gfs2 defaults 0 0
----------------------------

Do not forget to make this change on both (or, with DRBD 9.1, all) cluster nodes.

After this, you may mount your new filesystem by starting the
`gfs` service (on both nodes): indexterm:[GFS]

----------------------------
# service gfs start
----------------------------

From then onwards, as long as you have DRBD configured to start
automatically on system startup, before the RHCS services and the
`gfs` service, you will be able to use this GFS file system as you
would use one that is configured on traditional shared storage.
