ifdef::env-github[]
:tip-caption: :bulb:
:note-caption: :information_source:
:important-caption: :heavy_exclamation_mark:
:caution-caption: :fire:
:warning-caption: :warning:
endif::[]


[[ch-admin-manual]]
== Common administrative tasks

This chapter outlines typical administrative tasks encountered during
day-to-day operations. It does not cover troubleshooting tasks, these
are covered in detail in <<ch-troubleshooting>>.

include::configure.adoc[]

[[s-check-status]]
=== Checking DRBD status

[[s-drbd-drbdmon]]
==== Retrieving status with drbdmon

One convenient way to look at DRBD's status is the
`drbdmon` utility. It updates the state of DRBD resources in real time.

[[s-proc-drbd]]
==== Status information in `/proc/drbd`

NOTE: `/proc/drbd` is deprecated. While it won't be removed in the 8.4
series, we recommend to switch to other means, like <<s-drbdadm-status>>; or,
for monitoring even more convenient, <<s-drbdsetup-events2>>.

indexterm:[/proc/drbd]`/proc/drbd` is a virtual file displaying
basic information about the DRBD module.
It was used extensively up to DRBD 8.4, but couldn't keep up
with the amount of information provided by DRBD 9.

----------------------------
$ cat /proc/drbd
version: 9.0.0 (api:1/proto:86-110)
GIT-hash: XXX build by linbit@buildsystem.linbit, 2011-10-12 09:07:35
----------------------------

The first line, prefixed with `version:`, shows the DRBD version used
on your system. The second line contains information about this
specific build.


[[s-drbdadm-status]]
==== Status information via `drbdadm`

indexterm:[drbdadm, status]In its simplest invocation, we just ask for the
status of a single resource.

----------------------------
# drbdadm status home
home role:Secondary
  disk:UpToDate
  nina role:Secondary
    disk:UpToDate
  nino role:Secondary
    disk:UpToDate
  nono connection:Connecting
----------------------------

This here just says that the resource _home_ is locally, on 'nina', and
on 'nino' _UpToDate_ and _Secondary_; so the three nodes have the same
data on their storage devices, and nobody is using the device currently.

The node 'nono' is not connected, its state is reported as
_Connecting_; please see <<s-connection-states>> below for more details.


You can get more information by passing the `--verbose` and/or
`--statistics` arguments to `drbdsetup` (lines broken for readability):

----------------------------
# drbdsetup status home --verbose --statistics
home node-id:1 role:Secondary suspended:no
    write-ordering:none
  volume:0 minor:0 disk:UpToDate
      size:1048412 read:0 written:1048412 al-writes:0 bm-writes:48 upper-pending:0
                                        lower-pending:0 al-suspended:no blocked:no
  nina local:ipv4:10.9.9.111:7001 peer:ipv4:10.9.9.103:7010 node-id:0
                                               connection:Connected role:Secondary
      congested:no
    volume:0 replication:Connected disk:UpToDate resync-suspended:no
        received:1048412 sent:0 out-of-sync:0 pending:0 unacked:0
  nino local:ipv4:10.9.9.111:7021 peer:ipv4:10.9.9.129:7012 node-id:2
                                               connection:Connected role:Secondary
      congested:no
    volume:0 replication:Connected disk:UpToDate resync-suspended:no
        received:0 sent:0 out-of-sync:0 pending:0 unacked:0
  nono local:ipv4:10.9.9.111:7013 peer:ipv4:10.9.9.138:7031 node-id:3
                                                           connection:Connecting
----------------------------


Every few lines in this example form a block that is repeated
for every node used in this resource, with small format exceptions
for the local node - see below for more details.

The first line in each block shows the `node-id` (for the current
resource; a host can have different ``node-id``s in different resources).
Furthermore the `role` (see <<s-roles>>) is shown.

The next important line begins with the `volume` specification; normally
these are numbered starting by zero, but the configuration may specify
other IDs as well. This line shows the indexterm:[connection state]
connection state in the
`replication` item (see <<s-connection-states>> for details) and the
remote indexterm:[disk state] disk state in `disk` (see <<s-disk-states>>).
Then there's a line for this volume giving a bit of statistics -
data `received`, `sent`, `out-of-sync`, etc; please see
<<s-performance-indicators>> and <<s-conn-info>> for more information.

For the local node the first line shows the resource name, _home_, in our
example. As the first block always describes the local node, there is no _Connection_ or
address information.

please see the `drbd.conf` manual page for more information.

The other four lines in this example form a block that is repeated for
every DRBD device configured, prefixed by the device minor number. In
this case, this is `0`, corresponding to the device `/dev/drbd0`.

The resource-specific output contains various pieces
of information about the resource:

///////
//FIXME
.Replication protocol
The Replication protocol used by the resource. Either `A`, `B`, or `C`. See
<<s-replication-protocols>> for details.
///////

[[s-drbdsetup-events2]]
==== One-shot or real time monitoring via `drbdsetup events2`

NOTE: This is available only with userspace versions 8.9.3 and up.
// and 8.4.6

This is a low-level mechanism to get information out of DRBD, suitable for use
in automated tools, like monitoring.

In its simplest invocation, showing only the current status, the output looks
like this (but, when running on a terminal, will include colors):

---------------
# drbdsetup events2 --now r0
exists resource name:r0 role:Secondary suspended:no
exists connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connected role:Secondary
exists device name:r0 volume:0 minor:7 disk:UpToDate
exists device name:r0 volume:1 minor:8 disk:UpToDate
exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:0
    replication:Established peer-disk:UpToDate resync-suspended:no
exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:1
    replication:Established peer-disk:UpToDate resync-suspended:no
exists -
---------------

Without the ''--now'', the process will keep running, and send continuous updates like this:

-----------------
# drbdsetup events2 r0
...
change connection name:r0 peer-node-id:1 conn-name:remote-host connection:StandAlone
change connection name:r0 peer-node-id:1 conn-name:remote-host connection:Unconnected
change connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connecting
-----------------

Then, for monitoring purposes, there's another argument ''--statistics'', that
will produce some performance counters and other facts:

'drbdsetup' **verbose output** (lines broken for readability):

-----------------
# drbdsetup events2 --statistics --now r0
exists resource name:r0 role:Secondary suspended:no write-ordering:drain
exists connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connected
                                                        role:Secondary congested:no
exists device name:r0 volume:0 minor:7 disk:UpToDate size:6291228 read:6397188
            written:131844 al-writes:34 bm-writes:0 upper-pending:0 lower-pending:0
                                                         al-suspended:no blocked:no
exists device name:r0 volume:1 minor:8 disk:UpToDate size:104854364 read:5910680
          written:6634548 al-writes:417 bm-writes:0 upper-pending:0 lower-pending:0
                                                         al-suspended:no blocked:no
exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:0
          replication:Established peer-disk:UpToDate resync-suspended:no received:0
                                      sent:131844 out-of-sync:0 pending:0 unacked:0
exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:1
          replication:Established peer-disk:UpToDate resync-suspended:no received:0
                                     sent:6634548 out-of-sync:0 pending:0 unacked:0
exists -
-----------------

You might also like the ''--timestamp'' parameter.


[[s-connection-states]]
==== Connection states

indexterm:[drbdadm, cstate]indexterm:[connection state, via drbdadm]A resource's connection state can be
observed either by issuing the `drbdadm cstate` command:

----------------------------
# drbdadm cstate <resource>
Connected
Connected
StandAlone
----------------------------

If you are interested in only a single connection of a resource, specify the connection name, too:

The default is the peer's hostname as given in the configuration file.

----------------------------
# drbdadm cstate <peer>:<resource>
Connected
----------------------------

A resource may have one of the following connection states:

._StandAlone_
indexterm:[connection state, StandAlone]No network configuration available. The
resource has not yet been connected, or has been administratively
disconnected (using `drbdadm disconnect`), or has dropped its
connection due to failed authentication or split brain.

._Disconnecting_
indexterm:[connection state, Disconnecting]Temporary state during disconnection. The
next state is _StandAlone_.

._Unconnected_
indexterm:[connection state, Unconnected]Temporary state, prior to a connection
attempt. Possible next states: _Connecting_.

._Timeout_
indexterm:[connection state, Timeout]Temporary state following a timeout in the
communication with the peer. Next state: _Unconnected_.

._BrokenPipe_
indexterm:[connection state, BrokenPipe]Temporary state after the connection to
the peer was lost. Next state: _Unconnected_.

._NetworkFailure_
indexterm:[connection state, NetworkFailure]Temporary state after the connection to
the partner was lost. Next state: _Unconnected_.

._ProtocolError_
indexterm:[connection state, ProtocolError]Temporary state after the connection to
the partner was lost. Next state: _Unconnected_.

._TearDown_
indexterm:[connection state, TearDown]Temporary state. The peer is closing the
connection. Next state: _Unconnected_.

._Connecting_
indexterm:[connection state, Connecting]This node is waiting until the peer node
becomes visible on the network.

._Connected_
indexterm:[connection state, Connected]A DRBD connection has been established,
data mirroring is now active. This is the normal state.

[[s-replication-states]]
==== Replication states

Each volume has over each connection a replication state. The possible replication
states are:

._Off_
indexterm:[replication state, Off]The volume is not replicated over this
connection, since the connection is not _Connected_.

._Established_
indexterm:[replication state, Established]All writes to that volume are replicated
online. This is the normal state.

._StartingSyncS_
indexterm:[replication state, StartingSyncS]Full synchronization, initiated by the
administrator, is just starting. The next possible states are:
_SyncSource_ or _PausedSyncS_.

._StartingSyncT_
indexterm:[replication state, StartingSyncT]Full synchronization, initiated by the
administrator, is just starting. Next state: _WFSyncUUID_.

._WFBitMapS_
indexterm:[replication state, WFBitMapS]Partial synchronization is just
starting. Next possible states: _SyncSource_ or _PausedSyncS_.

._WFBitMapT_
indexterm:[replication state, WFBitMapT]Partial synchronization is just
starting. Next possible state: _WFSyncUUID_.

._WFSyncUUID_
indexterm:[replication state, WFSyncUUID]Synchronization is about to begin. Next
possible states: _SyncTarget_ or _PausedSyncT_.

._SyncSource_
indexterm:[replication state, SyncSource]Synchronization is currently running, with
the local node being the source of synchronization.

._SyncTarget_
indexterm:[replication state, SyncTarget]Synchronization is currently running, with
the local node being the target of synchronization.

._PausedSyncS_
indexterm:[replication state, PausedSyncS]The local node is the source of an ongoing
synchronization, but synchronization is currently paused. This may be
due to a dependency on the completion of another synchronization
process, or due to synchronization having been manually interrupted by
`drbdadm pause-sync`.

._PausedSyncT_
indexterm:[replication state, PausedSyncT]The local node is the target of an ongoing
synchronization, but synchronization is currently paused. This may be
due to a dependency on the completion of another synchronization
process, or due to synchronization having been manually interrupted by
`drbdadm pause-sync`.

._VerifyS_
indexterm:[replication state, VerifyS]On-line device verification is currently
running, with the local node being the source of verification.

._VerifyT_
indexterm:[replication state, VerifyT]On-line device verification is currently
running, with the local node being the target of verification.

._Ahead_
indexterm:[replication state, Ahead]Data replication was suspended, since the
link can not cope with the load. This state is enabled by the configuration
`on-congestion` option (see <<s-configure-congestion-policy>>).

._Behind_
indexterm:[replication state, Behind]Data replication was suspended by the peer,
since the link can not cope with the load. This state is enabled by the configuration
`on-congestion` option on the peer node (see <<s-configure-congestion-policy>>).

[[s-roles]]
==== Resource roles

indexterm:[resource]A resource's role can be observed by issuing the
indexterm:[drbdadm, role]`drbdadm role` command:

----------------------------
# drbdadm role <resource>
Primary
----------------------------

You may see one of the following resource roles:

._Primary_
indexterm:[role, Primary]The resource is currently in the primary role, and may be read from
and written to. This role only occurs on one of the two nodes, unless
<<s-dual-primary-mode,dual-primary mode>> is enabled.

._Secondary_
indexterm:[role, Secondary]The resource is currently in the secondary role. It normally receives
updates from its peer (unless running in disconnected mode), but may
neither be read from nor written to. This role may occur on one
or both nodes.

._Unknown_
indexterm:[role, Unknown]The resource's role is currently unknown. The local resource role
never has this status. It is only displayed for the peer's resource
role, and only in disconnected mode.


[[s-disk-states]]
==== Disk states

A resource's disk state can be observed either by
issuing the `drbdadm dstate` command:

----------------------------
# drbdadm dstate <resource>
UpToDate
----------------------------

The disk state may be one of the following:

._Diskless_
indexterm:[disk state, Diskless]No local block device has been assigned to the
DRBD driver. This may mean that the resource has never attached to its
backing device, that it has been manually detached using `drbdadm
detach`, or that it automatically detached after a lower-level I/O
error.

._Attaching_
indexterm:[disk state, Attaching]Transient state while reading meta data.

._Detaching_
indexterm:[disk state, Detaching]Transient state while detaching and waiting
for ongoing IOs to complete.

._Failed_
indexterm:[disk state, Failed]Transient state following an I/O failure report
by the local block device. Next state: _Diskless_.

._Negotiating_
indexterm:[disk state, Negotiating]Transient state when an _Attach_ is carried out on
an already-__Connected__ DRBD device.

._Inconsistent_
indexterm:[disk state, Inconsistent]The data is inconsistent. This status occurs
immediately upon creation of a new resource, on both nodes (before the
initial full sync). Also, this status is found in one node (the
synchronization target) during synchronization.

._Outdated_
indexterm:[disk state, Outdated]Resource data is consistent, but
<<s-outdate,outdated>>.

._DUnknown_
indexterm:[disk state, DUnknown]This state is used for the peer disk if no
network connection is available.

._Consistent_
indexterm:[disk state, Consistent]Consistent data of a node without
connection. When the connection is established, it is decided whether
the data is _UpToDate_ or _Outdated_.

._UpToDate_
indexterm:[disk state, UpToDate]Consistent, up-to-date state of the data. This
is the normal state.


[[s-conn-info]]
==== Connection information data

._local_
Shows the network family, the local address and port that is used to
accept connections from the peer.

._peer_
Shows the network family, the peer address and port that is used
to connect.

._congested_
This flag tells whether the TCP send buffer of the data connection is more than 80% filled.


[[s-performance-indicators]]
==== Performance indicators

indexterm:[performance indicators]
The command ``drbdsetup status --verbose --statistics`` can be used to show performance statistics.
These are also available in ``drbdsetup events2 --statistics``,
although there will not be a _changed_ event for every change.
The statistics include the following counters and gauges:

Per volume/device:

._read_ (disk read)
Net data read from local disk; in KiB.

._written_ (disk written)
Net data written to local disk; in KiB.

._al-writes_ (activity log)
Number of updates of the activity log area of the meta data.

._bm-writes_ (bitmap)
Number of updates of the bitmap area of the meta data.

._upper-pending_ (application pending)
Number of block I/O requests forwarded to DRBD, but not yet answered (completed)
by DRBD.

._lower-pending_ (local count)
Number of open requests to the local I/O sub-system issued by DRBD.

._blocked_
indexterm:[blocked]Shows local I/O congestion.
--

  * _no_: No congestion.
  * _upper_: I/O _above_ the DRBD device is blocked, i.e. to the filesystem. Typical causes are
    ** I/O suspension by the administrator, see the `suspend-io` command in `drbdadm`.
    ** transient blocks, e.g. during attach/detach
    ** buffers depleted, see <<p-performance>>
    ** Waiting for bitmap IO
  * _lower_: Backing device is congested.
  * _upper,lower_: Both _upper_ and _lower_ are blocked.
--

Per connection:

._ap-in-flight_ (application in-flight)
Application data that is being written by the peer.
That is, DRBD has sent it to the peer and is waiting for the acknowledgement that it has been written.
In sectors (512 bytes).

._rs-in-flight_ (resync in-flight)
Resync data that is being written by the peer.
That is, DRBD is _SyncSource_, has sent data to the peer as part of a resync
and is waiting for the acknowledgement that it has been written.
In sectors (512 bytes).

Per connection and volume ("peer device"):

._received_ (network receive)
Net data received from the peer; in KiB.

._sent_ (network send)
Net data sent to the peer; in KiB.

._out-of-sync_
indexterm:[out-of-sync]indexterm:[oos]Amount of data currently out of sync
with this peer, according to the bitmap that DRBD has for it; in KiB.

._pending_
Number of requests sent to the peer, but that have not yet been
acknowledged by the peer.

._unacked_ (unacknowledged)
Number of requests received from the peer,
but that have not yet been acknowledged by DRBD on this node.

._resync-suspended_
Whether the resynchronization is currently suspended or not. Possible values are _no_, _user_, _peer_, _dependency_.
Comma separated.


[[s-enable-disable]]
=== Enabling and disabling resources

[[s-enable-resource]]
==== Enabling resources

indexterm:[resource]Normally, all configured DRBD resources are
automatically enabled

* by a cluster resource management application at its discretion,
  based on your cluster configuration, or

* by the systemd units (e.g., `drbd@.target`)

If, however, you need to enable resources manually for any reason, you
may do so by issuing the command

----------------------------
# drbdadm up <resource>
----------------------------

As always, you may use the keyword `all` instead of a specific
resource name if you want to enable all resources configured in
`/etc/drbd.conf` at once.

[[s-disable-resource]]
==== Disabling resources

indexterm:[resource]You may temporarily disable specific resources by
issuing the command

----------------------------
# drbdadm down <resource>
----------------------------

Here, too, you may use the keyword `all` in place of a resource name if
you wish to temporarily disable all resources listed in
`/etc/drbd.conf` at once.

[[s-reconfigure]]
=== Reconfiguring resources

indexterm:[resource]DRBD allows you to reconfigure resources while
they are operational. To that end,

* make any necessary changes to the resource configuration in
  `/etc/drbd.conf`,

* synchronize your `/etc/drbd.conf` file between both nodes,

* issue the indexterm:[drbdadm, adjust]`drbdadm adjust <resource>` command on
  both nodes.

`drbdadm adjust` then hands off to `drbdsetup` to make the necessary
adjustments to the configuration. As always, you are able to review
the pending `drbdsetup` invocations by running `drbdadm` with the
`-d` (dry-run) option.

NOTE: When making changes to the `common` section in `/etc/drbd.conf`,
you can adjust the configuration for all resources in one run, by
issuing `drbdadm adjust all`.

[[s-switch-resource-roles]]
=== Promoting and demoting resources

indexterm:[resource]Manually switching a <<s-resource-roles,resource's
role>> from secondary to primary (promotion) or vice versa (demotion)
is done using one of the following commands:

----------------------------
# drbdadm primary <resource>
# drbdadm secondary <resource>
----------------------------

In <<s-single-primary-mode,single-primary mode>> (DRBD's default), any
resource can be in the primary role on only one node at any given time
while the <<s-connection-states,connection state>> is
_Connected_. Thus, issuing `drbdadm primary <resource>` on one node
while the specified resource is still in the primary role on another node will
result in an error.

A resource configured to allow <<s-dual-primary-mode,dual-primary
mode>> can be switched to the primary role on two nodes; this is e.g. needed for
online migration of virtual machines.


[[s-manual-fail-over]]
=== Basic manual fail-over

If not using a cluster manager and looking to handle fail-overs manually in a
passive/active configuration, the process is as follows.

On the current primary node stop any applications or services using the DRBD device,
unmount the DRBD device, and demote the resource to secondary.

----------------------------
# umount /dev/drbd/by-res/<resource>/<vol-nr>
# drbdadm secondary <resource>
----------------------------

Now on the node we wish to make primary promote the resource and mount the device.

----------------------------
# drbdadm primary <resource>
# mount /dev/drbd/by-res/<resource>/<vol-nr> <mountpoint>
----------------------------

If you're using the `auto-promote` feature, you don't need to change the roles
(_Primary_/_Secondary_) manually; only stopping of the services and unmounting,
respectively mounting, is necessary.

[[s-upgrading-drbd]]
include::upgrading.adoc[]

[[s-enable-dual-primary]]
include::dualprimary.adoc[]

[[s-use-online-verify]]
=== Using on-line device verification

[[s-online-verify-enable]]
==== Enabling on-line verification

indexterm:[on-line device verification]<<s-online-verify,On-line
device verification>> for resources is not enabled by default. To
enable it, add the following lines to your resource configuration in
`/etc/drbd.conf`:

[source,drbd]
----------------------------
resource <resource>
  net {
    verify-alg <algorithm>;
  }
  ...
}
----------------------------

_<algorithm>_ may be any message digest algorithm supported by the
kernel crypto API in your system's kernel configuration. Normally, you
should be able to choose at least from `sha1`, `md5`, and `crc32c`.

If you make this change to an existing resource, as always,
synchronize your `drbd.conf` to the peer, and run `drbdadm adjust
_<resource>_` on both nodes.

[[s-online-verify-invoke]]
==== Invoking on-line verification

indexterm:[on-line device verification]After you have enabled on-line
verification, you will be able to initiate a verification run using
the following command:

----------------------------
# drbdadm verify <resource>:<peer>/<volume>
----------------------------

When you do so, DRBD starts an online verification run for
_<volume>_ to _<peer_> in _<resource>_, and if it detects any blocks
that are not in sync, will mark those blocks as such and write a
message to the kernel log. Any applications using the device at that
time can continue to do so unimpeded, and you may also
<<s-switch-resource-roles,switch resource roles>> at will.

_<Volume>_ is optional, if ommited, it will verify all volumes
in that resource.

If out-of-sync blocks were detected during the verification run, you
may resynchronize them using the following commands after verification
has completed.

Since drbd-9.0.29 the preferred way is one of these commands:

----------------------------
# drbdadm invalidate <resource>:<peer>/volume --reset-bitmap=no
# drbdadm invalidate-remote <resource>:<peer>/volume --reset-bitmap=no
----------------------------

The first command will cause the local differences to be overwritten
by the remote version. The second command does it in the opposite direction.

Before drbd-9.0.29 one needs to initiate a resync. A way to do that
is disconnecting from a primary and ensuring that the primary changes
at least one block while the peer is away.

----------------------------
# drbdadm disconnect <resource>:<peer>
## write one block on the primary
# drbdadm connect <resource>:<peer>
----------------------------


[[s-online-verify-automate]]
==== Automating on-line verification

indexterm:[on-line device verification]Most users will want to
automate on-line device verification. This can be easily
accomplished. Create a file with the following contents, named
`/etc/cron.d/drbd-verify` on *one* of your nodes:

[source,drbd]
----------------------------
42 0 * * 0    root    /sbin/drbdadm verify <resource>
----------------------------

This will have `cron` invoke a device verification every Sunday at 42
minutes past midnight; so, if you come into the office on Monday morning,
a quick look at the resource's status would show the result. If your devices
are very big, and the \~32 hours were not enough, then you'll notice _VerifyS_ or
_VerifyT_ as connection state, meaning that the `verify` is still in progress.

If you have enabled on-line verification for all your resources (for
example, by adding `verify-alg _<algorithm>_` to the `common` section
in `/etc/drbd.d/global_common.conf`), you may also use:

[source,drbd]
----------------------------
42 0 * * 0    root    /sbin/drbdadm verify all
----------------------------


[[s-configure-sync-rate]]
=== Configuring the rate of synchronization

indexterm:[synchronization]Normally, one tries to ensure that
background synchronization (which makes the data on the
synchronization target temporarily inconsistent) completes as quickly
as possible. However, it is also necessary to keep background
synchronization from hogging all bandwidth otherwise available for
foreground replication, which would be detrimental to application
performance. Thus, you must configure the synchronization bandwidth to
match your hardware -- which you may do in a permanent fashion or
on-the-fly.

IMPORTANT: It does not make sense to set a synchronization rate that
is higher than the maximum write throughput on your secondary
node. You must not expect your secondary node to miraculously be able
to write faster than its I/O subsystem allows, just because it happens
to be the target of an ongoing device synchronization.

Likewise, and for the same reasons, it does not make sense to set a
synchronization rate that is higher than the bandwidth available on
the replication network.


[[s-estimating_a_synchronization_speed]]
==== Estimating a synchronization speed

TIP: A good rule of thumb for this value is to use about 30% of the
available replication bandwidth. Thus, if you had an I/O subsystem
capable of sustaining write throughput of 400MB/s, and a Gigabit
Ethernet network capable of sustaining 110 MB/s network throughput
(the network being the bottleneck), you would calculate:

[[eq-sync-rate-example1]]
.Syncer rate example, 110MB/s effective available bandwidth
image::images/sync-rate-example1.svg[]

Thus, the recommended value for the `rate` option would be `33M`.

By contrast, if you had an I/O subsystem with a maximum throughput of
80MB/s and a Gigabit Ethernet connection (the I/O subsystem being the
bottleneck), you would calculate:

[[eq-sync-rate-example2]]
.Syncer rate example, 80MB/s effective available bandwidth
image::images/sync-rate-example2.svg[]

In this case, the recommended value for the `rate` option would be
`24M`.

Similarly, for a storage speed of 800MB/s and a 10Gbe network connection, you
would shoot for \~240MB/s synchronization rate.

[[s-configure-sync-rate-variable]]
==== Variable sync rate configuration

When multiple DRBD resources share a
single replication/synchronization network, synchronization with a fixed rate
may not be an optimal approach. So, in DRBD 8.4.0 the
variable-rate synchronization was enabled by default. In this mode, DRBD uses an automated
control loop algorithm to determine, and adjust, the
synchronization rate. This algorithm ensures that there is always
sufficient bandwidth available for foreground replication, greatly
mitigating the impact that background synchronization has on
foreground I/O.

The optimal configuration for variable-rate synchronization may vary
greatly depending on the available network bandwidth, application I/O
pattern and link congestion.
ifndef::drbd-only,de-brand[]
Ideal configuration settings also depend
on whether <<s-drbd-proxy,DRBD Proxy>> is in use or not.
It may be
wise to engage professional consultancy in order to optimally
configure this DRBD feature. An _example_ configuration (which assumes
a deployment in conjunction with DRBD Proxy) is provided below:

[source,drbd]
----------------------------
resource <resource> {
  disk {
    c-plan-ahead 5;
    c-max-rate 10M;
    c-fill-target 2M;
  }
}
----------------------------

endif::[]

TIP: A good starting value for `c-fill-target` is _BDP * 2_, where
_BDP_ is your bandwidth-delay-product on the replication link.

For example, when using a 1GBit/s crossover connection, you'll end up with
about 200µs latencyfootnote:[The rule-of-thumb is using the time reported by
`ping`.]. +
1GBit/s means about 120MB/s; times 200*10^-6^ seconds gives 24000 Byte. Just round that value up to the next MB, and you're good to go.

Another example: a 100MBit WAN connection with 200ms latency means 12MB/s times
0.2s, or about 2.5MB "on the wire". Here a good starting value for
`c-fill-target` would be 3MB.

Please see the `drbd.conf` manual page for more details on the other
configuration items.


[[s-configure-sync-rate-permanent]]
==== Permanent fixed sync rate configuration

In a few, very restricted situationsfootnote:[Like benchmarking.], it might
make sense to just use some fixed synchronization rate. In this case, first
of all you need to turn the dynamic sync rate controller off, by using
`c-plan-ahead 0;`.

Then, the maximum bandwidth a resource uses for background
re-synchronization is determined by the `resync-rate` option
for a resource. This must be included in the resource's
`disk` section in `/etc/drbd.conf`:

[source,drbd]
----------------------------
resource <resource>
  disk {
    resync-rate 40M;
    ...
  }
  ...
}
----------------------------

Note that the rate setting is given in _bytes_, not _bits_ per second; the
default unit is _Kibibyte_, so a value of `4096` would be interpreted as `4MiB`.

NOTE: This just defines a rate that DRBD tries to achieve. If there is a
bottleneck with lower throughput (network, storage speed), the defined speed
(aka the "wished-for" performance ;) won't be reached.


[[s-some_more_hints_about_synchronization]]
==== Some more hints about synchronization

When some amount of the to-be-synchronized data isn't really in use anymore
(e.g. because files got deleted while one node wasn't connected), you
might benefit from the <<s-trim-discard>>.

Furthermore, `c-min-rate` is easy to misunderstand - it doesn't define
a *minimum* synchronization speed, but rather a limit below which DRBD will not
slow down further *on purpose*. +
Whether you manage to *reach* that synchronization rate depends on your network
and storage speed, network latency (which might be highly variable for shared
links), and application IO (which not might be able to do anything about).


[[s-configure-checksum-sync]]
=== Configuring checksum-based synchronization

indexterm:[checksum-based
synchronization]<<p-checksum-sync,Checksum-based synchronization>> is
not enabled for resources by default. To enable it, add the following
lines to your resource configuration in `/etc/drbd.conf`:

[source,drbd]
----------------------------
resource <resource>
  net {
    csums-alg <algorithm>;
  }
  ...
}
----------------------------

_<algorithm>_ may be any message digest algorithm supported by the
kernel crypto API in your system's kernel configuration. Normally, you
should be able to choose at least from `sha1`, `md5`, and `crc32c`.

If you make this change to an existing resource, as always,
synchronize your `drbd.conf` to the peer, and run `drbdadm adjust
_<resource>_` on both nodes.

[[s-configure-congestion-policy]]
=== Configuring congestion policies and suspended replication

In an environment where the replication bandwidth is highly variable
(as would be typical in WAN replication setups), the replication link
may occasionally become congested. In a default configuration, this
would cause I/O on the primary node to block, which is sometimes
undesirable.

Instead, you may configure DRBD to _suspend_ the ongoing replication
in this case, causing the Primary's data set to _pull ahead_ of the
Secondary. In this mode, DRBD keeps the replication channel open -- it
never switches to disconnected mode -- but does not actually replicate
until sufficient bandwidth becomes available again.

The following example is for a DRBD Proxy configuration:

[source,drbd]
----------------------------
resource <resource> {
  net {
    on-congestion pull-ahead;
    congestion-fill 2G;
    congestion-extents 2000;
    ...
  }
  ...
}
----------------------------

It is usually wise to set both `congestion-fill` and
`congestion-extents` together with the `pull-ahead` option.

A good value for `congestion-fill` is 90%

* of the allocated DRBD proxy buffer memory, when replicating over
  DRBD Proxy, or
* of the TCP network send buffer, in non-DRBD Proxy setups.

A good value for `congestion-extents` is 90% of your configured
`al-extents` for the affected resources.


[[s-configure-io-error-behavior]]
=== Configuring I/O error handling strategies

indexterm:[I/O errors]indexterm:[drbd.conf]DRBD's
<<s-handling-disk-errors,strategy for handling lower-level I/O
errors>> is determined by the `on-io-error` option, included in the
resource `disk` configuration in `/etc/drbd.conf`:

[source,drbd]
----------------------------
resource <resource> {
  disk {
    on-io-error <strategy>;
    ...
  }
  ...
}
----------------------------

You may, of course, set this in the `common` section too, if you want
to define a global I/O error handling policy for all resources.

_<strategy>_ may be one of the following options:

.`detach`
This is the default and recommended option. On the occurrence of a
lower-level I/O error, the node drops its backing device, and
continues in diskless mode.

.`pass-on`
This causes DRBD to report the I/O error to the upper layers. On the
primary node, it is reported to the mounted file system. On the
secondary node, it is ignored (because the secondary has no upper
layer to report to).

.`call-local-io-error`
Invokes the command defined as the local I/O error handler. This
requires that a corresponding `local-io-error` command invocation is
defined in the resource's `handlers` section. It is entirely left to
the administrator's discretion to implement I/O error handling using
the command (or script) invoked by `local-io-error`.

NOTE: Early DRBD versions (prior to 8.0) included another option,
`panic`, which would forcibly remove the node from the cluster by way
of a kernel panic, whenever a local I/O error occurred. While that
option is no longer available, the same behavior may be mimicked via
the `local-io-error`/`call-local-io-error` interface. You should do so
only if you fully understand the implications of such behavior.


You may reconfigure a running resource's I/O error handling strategy
by following this process:

  * Edit the resource configuration in `/etc/drbd.d/__<resource>__.res`.

  * Copy the configuration to the peer node.

  * Issue `drbdadm adjust _<resource>_` on both nodes.


[[s-configure-integrity-check]]
=== Configuring replication traffic integrity checking

indexterm:[replication traffic integrity
checking]<<s-integrity-check,Replication traffic integrity checking>>
is not enabled for resources by default. To enable it, add the
following lines to your resource configuration in `/etc/drbd.conf`:

[source,drbd]
----------------------------
resource <resource>
  net {
    data-integrity-alg <algorithm>;
  }
  ...
}
----------------------------

_<algorithm>_ may be any message digest algorithm supported by the
kernel crypto API in your system's kernel configuration. Normally, you
should be able to choose at least from `sha1`, `md5`, and `crc32c`.

If you make this change to an existing resource, as always,
synchronize your `drbd.conf` to the peer, and run `drbdadm adjust
_<resource>_` on both nodes.

WARNING: This feature is not intended for production use. Enable only if you need to
diagnose data corruption problems, and want to see whether the transport path
(network hardware, drivers, switches) might be at fault!

[[s-resizing]]
=== Resizing resources


When growing (extending) DRBD volumes, you need to grow from bottom to top.
You need to extend the backing block devices on all nodes first.
Then you can tell DRBD to use the new space.

Once the DRBD volume is extended, you need to still propagate that
into whatever is using DRBD: extend the file system, or make a VM
running with this volume attached aware of the new "disk size".

That all typically boils down to

-----------------------
# # on all nodes, resize the backing LV:
# lvextend -L +${additional_gb}g VG/LV
# # on one node:
# drbdadm resize ${resource_name}/${volume_number}
# # on the Primary only:
# # resize the file system using the file system specific tool, see below
-----------------------

See also the next sections <<s-growing-online>> and following.

Note that different file systems have different capabilities and different sets
of management tools. For example XFS can only grow.
You point its tool to the active mount point: `xfs_growfs /where/you/have/it/mounted`.

While the EXT family can both grow (even online), and also shrink (only
offline; you have to unmount it first). To resize an ext3 or ext4,
you would point the tool not to the mount point, but to the (mounted)
block device: `resize2fs /dev/drbd#`

Obviously use the correct DRBD (as displayed by `mount` or `df -T`, while mounted),
and *not* the backing block device.  As long as DRBD is up, that's not supposed
to work anyways (`resize2fs: Device or resource busy while trying to open
/dev/mapper/VG-LV Couldn't find valid filesystem superblock.`).
If you tried to do that offline (with DRBD stopped), you may corrupt DRBD
meta data if you ran the file system tools directly against the backing LV or
partition.  So don't.

You do the file system resize only *once* on the Primary, against the active
DRBD device.  DRBD replicates the changes to the file system structure.
That is what you have it for.

Also, don't use resize2fs on XFS volumes, or XFS tools on EXT, or ...
but the right tool for the file system in use.

`resize2fs: Bad magic number in super-block while trying to open /dev/drbd7`
is probably just trying to tell you that this is *not* an EXT file system,
and you should try an other tool instead. Maybe `xfs_growfs`? But as mentioned,
that does not take the block device, but the mount point as argument.

When shrinking (reducing) DRBD volumes, you need to shrink from top to bottom.
So first make sure no one is using the space you want to cut off.
Next, shrink the file system (if your file system supports that).
Then tell DRBD to stop using that space, which is not so easy with DRBD
internal meta data, because they are by design "at the end"
of the backing device.

Once you are sure that DRBD won't use the space anymore either,
you can cut it off from the backing device, for example using `lvreduce`.

See also <<s-shrinking-online>>, <<s-shrinking-offline>>.


[[s-growing-online]]
==== Growing on-line

indexterm:[resource]If the backing block devices can be grown while in
operation (online), it is also possible to increase the size of a DRBD
device based on these devices during operation. To do so, two criteria
must be fulfilled:

. The affected resource's backing device must be one managed by a
  logical volume management subsystem, such as LVM.

. The resource must currently be in the _Connected_ connection state.

Having grown the backing block devices on all nodes, ensure that only
one node is in primary state. Then enter on one node:

----------------------------
# drbdadm resize <resource>
----------------------------

This triggers a synchronization of the new section. The
synchronization is done from the primary node to the secondary node.

If the space you're adding is clean, you can skip syncing the additional
space by using the --assume-clean option.

----------------------------
# drbdadm -- --assume-clean resize <resource>
----------------------------

[[s-growing-offline]]
==== Growing off-line

indexterm:[resource]When the backing block devices on both nodes are
grown while DRBD is inactive, and the DRBD resource is using
<<s-external-meta-data,external meta data>>, then the new size is
recognized automatically. No administrative intervention is
necessary. The DRBD device will have the new size after the next
activation of DRBD on both nodes and a successful establishment of a
network connection.

If however the DRBD resource is configured to use
<<s-internal-meta-data,internal meta data>>, then this meta data must
be moved to the end of the grown device before the new size becomes
available. To do so, complete the following steps:

WARNING: This is an advanced procedure. Use at your own discretion.

* Unconfigure your DRBD resource:

[source,drbd]

----------------------------
# drbdadm down <resource>
----------------------------

* Save the meta data in a text file prior to resizing:

----------------------------
# drbdadm dump-md <resource> > /tmp/metadata
----------------------------

You must do this on both nodes, using a separate dump file for every
node. *Do not* dump the meta data on one node, and simply copy the
dump file to the peer. *This. will. not. work.*

* Grow the backing block device on both nodes.

* Adjust the size information (`la-size-sect`) in the file
  `/tmp/metadata` accordingly, on both nodes. Remember that
  `la-size-sect` must be specified in sectors.

* Re-initialize the metadata area:

----------------------------
# drbdadm create-md <resource>
----------------------------

* Re-import the corrected meta data, on both nodes:

----------------------------
# drbdmeta_cmd=$(drbdadm -d dump-md <resource>)
# ${drbdmeta_cmd/dump-md/restore-md} /tmp/metadata
Valid meta-data in place, overwrite? [need to type 'yes' to confirm]
yes
Successfully restored meta data
----------------------------

NOTE: This example uses `bash` parameter substitution. It may or may
not work in other shells. Check your `SHELL` environment variable if
you are unsure which shell you are currently using.

* Re-enable your DRBD resource:

----------------------------
# drbdadm up <resource>
----------------------------

* On one node, promote the DRBD resource:

----------------------------
# drbdadm primary <resource>
----------------------------

* Finally, grow the file system so it fills the extended size of the
  DRBD device.


[[s-shrinking-online]]
==== Shrinking on-line


WARNING: Online shrinking is only supported with external metadata.

indexterm:[resource]Before shrinking a DRBD device, you _must_ shrink
the layers above DRBD, i.e. usually the file system. Since DRBD cannot
ask the file system how much space it actually uses, you have to be
careful in order not to cause data loss.

NOTE: Whether or not the _filesystem_ can be shrunk on-line depends on
the filesystem being used. Most filesystems do not support on-line
shrinking. XFS does not support shrinking at all.

To shrink DRBD on-line, issue the following command _after_ you have
shrunk the file system residing on top of it:

[source,drbd]
----------------------------
# drbdadm resize --size=<new-size> <resource>
----------------------------

You may use the usual multiplier suffixes for _<new-size>_ (K, M, G
etc.). After you have shrunk DRBD, you may also shrink the containing
block device (if it supports shrinking).

NOTE: It might be a good idea to issue `drbdadm resize _<resource>_` after
resizing the lower level device, so that the DRBD metadata *really* gets
written into the expected space at the end of the volume.

[[s-shrinking-offline]]
==== Shrinking off-line

indexterm:[resource]If you were to shrink a backing block device while
DRBD is inactive, DRBD would refuse to attach to this block device
during the next attach attempt, since it is now too small (in case
external meta data is used), or it would be unable to find its meta
data (in case internal meta data is used). To work around these
issues, use this procedure (if you cannot use
<<s-shrinking-online,on-line shrinking>>):


WARNING: This is an advanced procedure. Use at your own discretion.

* Shrink the file system from one node, while DRBD is still
  configured.

* Unconfigure your DRBD resource:

----------------------------
# drbdadm down <resource>
----------------------------

* Save the meta data in a text file prior to shrinking:

----------------------------
# drbdadm dump-md <resource> > /tmp/metadata
----------------------------

You must do this on both nodes, using a separate dump file for every
node. *Do not* dump the meta data on one node, and simply copy the dump
file to the peer. *This. will. not. work.*

* Shrink the backing block device on both nodes.

* Adjust the size information (`la-size-sect`) in the file
  `/tmp/metadata` accordingly, on both nodes. Remember that
  `la-size-sect` must be specified in sectors.

* *Only if you are using internal metadata* (which at this time have
  probably been lost due to the shrinking process), re-initialize the
  metadata area:
+
----------------------------
# drbdadm create-md <resource>
----------------------------

* Re-import the corrected meta data, on both nodes:
+
----------------------------
# drbdmeta_cmd=$(drbdadm -d dump-md <resource>)
# ${drbdmeta_cmd/dump-md/restore-md} /tmp/metadata
Valid meta-data in place, overwrite? [need to type 'yes' to confirm]
yes
Successfully restored meta data
----------------------------

NOTE: This example uses `bash` parameter substitution. It may or may not
work in other shells. Check your `SHELL` environment variable if you
are unsure which shell you are currently using.

* Re-enable your DRBD resource:
+
----------------------------
# drbdadm up <resource>
----------------------------


[[s-disable-flushes]]
=== Disabling backing device flushes

CAUTION: You should only disable device flushes when running DRBD on
devices with a battery-backed write cache (BBWC). Most storage
controllers allow to automatically disable the write cache when the
battery is depleted, switching to write-through mode when the battery
dies. It is strongly recommended to enable such a feature.

Disabling DRBD's flushes when running without BBWC, or on BBWC with a
depleted battery, is _likely to cause data loss_ and should not be
attempted.

DRBD allows you to enable and disable <<s-disk-flush-support,backing
device flushes>> separately for the replicated data set and DRBD's own
meta data. Both of these options are enabled by default. If you wish
to disable either (or both), you would set this in the `disk` section
for the DRBD configuration file, `/etc/drbd.conf`.

To disable disk flushes for the replicated data set, include the
following line in your configuration:

[source,drbd]
----------------------------
resource <resource>
  disk {
    disk-flushes no;
    ...
  }
  ...
}
----------------------------


To disable disk flushes on DRBD's meta data, include the following
line:

[source,drbd]
----------------------------
resource <resource>
  disk {
    md-flushes no;
    ...
  }
  ...
}
----------------------------

After you have modified your resource configuration (and synchronized
your `/etc/drbd.conf` between nodes, of course), you may enable these
settings by issuing this command on both nodes:

----------------------------
# drbdadm adjust <resource>
----------------------------


In case only one of the serves has a BBWCfootnote:[For example, in the DR site
you might be using different hardware, right?], you should move the setting into a host section, like this:

[source,drbd]
----------------------------
resource <resource> {
  disk {
    ... common settings ...
  }

  on host-1 {
    disk {
      md-flushes no;
    }
    ...
  }
  ...
}
----------------------------


[[s-configure-split-brain-behavior]]
=== Configuring split brain behavior

[[s-split-brain-notification]]
==== Split brain notification

DRBD invokes the `split-brain` handler, if configured, at any time
split brain is _detected_. To configure this handler, add the
following item to your resource configuration:

----------------------------
resource <resource>
  handlers {
    split-brain <handler>;
    ...
  }
  ...
}
----------------------------

_<handler>_ may be any executable present on the system.

The DRBD distribution contains a split brain handler script that
installs as `/usr/lib/drbd/notify-split-brain.sh`. It simply sends a
notification e-mail message to a specified address. To configure the
handler to send a message to `root@localhost` (which is expected to be
an email address that forwards the notification to a real system
administrator), configure the `split-brain handler` as follows:

----------------------------
resource <resource>
  handlers {
    split-brain "/usr/lib/drbd/notify-split-brain.sh root";
    ...
  }
  ...
}
----------------------------

After you have made this modification on a running resource (and
synchronized the configuration file between nodes), no additional
intervention is needed to enable the handler. DRBD will simply invoke
the newly-configured handler on the next occurrence of split brain.

[[s-automatic-split-brain-recovery-configuration]]
==== Automatic split brain recovery policies

CAUTION: Configuring DRBD to automatically resolve data divergence
situations resulting from split-brain (or other) scenarios
is configuring for potential *automatic data loss*.
Understand the implications, and don't do it if you don't mean to.

TIP: You rather want to look into fencing policies, quorum settings,
cluster manager integration, and redundant cluster manager communication links
to *avoid* data divergence in the first place.

In order to be able to enable and configure DRBD's automatic split
brain recovery policies, you must understand that DRBD offers several
configuration options for this purpose. DRBD applies its split brain
recovery procedures based on the number of nodes in the Primary role
at the time the split brain is detected. To that end, DRBD examines
the following keywords, all found in the resource's `net` configuration
section:

.`after-sb-0pri`
Split brain has just been detected, but at this time the resource is
not in the Primary role on any host. For this option, DRBD understands
the following keywords:

* `disconnect`: Do not recover automatically, simply invoke the
  `split-brain` handler script (if configured), drop the connection and
  continue in disconnected mode.

* `discard-younger-primary`: Discard and roll back the modifications
  made on the host which assumed the Primary role last.

* `discard-least-changes`: Discard and roll back the modifications on
  the host where fewer changes occurred.

* `discard-zero-changes`: If there is any host on which no changes
  occurred at all, simply apply all modifications made on the other
  and continue.

.`after-sb-1pri`
Split brain has just been detected, and at this time the resource is
in the Primary role on one host. For this option, DRBD understands the
following keywords:

* `disconnect`: As with `after-sb-0pri`, simply invoke the
  `split-brain` handler script (if configured), drop the connection
  and continue in disconnected mode.

* `consensus`: Apply the same recovery policies as specified in
  `after-sb-0pri`. If a split brain victim can be selected after
  applying these policies, automatically resolve. Otherwise, behave
  exactly as if `disconnect` were specified.

* `call-pri-lost-after-sb`: Apply the recovery policies as specified
  in `after-sb-0pri`. If a split brain victim can be selected after
  applying these policies, invoke the `pri-lost-after-sb` handler on
  the victim node. This handler must be configured in the
  `handlers` section and is expected to forcibly remove the node from
  the cluster.

* `discard-secondary`: Whichever host is currently in the Secondary
  role, make that host the split brain victim.

.`after-sb-2pri`
Split brain has just been detected, and at this time the resource is
in the Primary role on both hosts. This option accepts the same
keywords as `after-sb-1pri` except `discard-secondary` and `consensus`.

NOTE: DRBD understands additional keywords for these three options,
which have been omitted here because they are very rarely used. Refer
to the man page of `drbd.conf` for details on split brain recovery keywords not
discussed here.

For example, a resource which serves as the block device for a GFS or
OCFS2 file system in dual-Primary mode may have its recovery policy
defined as follows:

----------------------------
resource <resource> {
  handlers {
    split-brain "/usr/lib/drbd/notify-split-brain.sh root"
    ...
  }
  net {
    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;
    ...
  }
  ...
}
----------------------------


[[s-three-nodes]]
=== Creating a stacked three-node setup

A three-node setup involves one DRBD device _stacked_ atop another.

NOTE: Stacking is deprecated in DRBD version
9.x, as more nodes can be implemented on a single level. See
<<s-drbdconf-conns>> for details.

[[s-stacking-considerations]]
==== Device stacking considerations

The following considerations apply to this type of setup:

* The stacked device is the active one. Assume you have configured one
  DRBD device `/dev/drbd0`, and the stacked device atop it is
  `/dev/drbd10`, then `/dev/drbd10` will be the device that you mount
  and use.

* Device meta data will be stored twice, on the underlying DRBD device
  _and_ the stacked DRBD device. On the stacked device, you must always
  use <<s-internal-meta-data,internal meta data>>. This means that the
  effectively available storage area on a stacked device is slightly
  smaller, compared to an unstacked device.

* To get the stacked upper level device running, the underlying device
  must be in the primary role.

* To be able to synchronize the backup node, the stacked device on the
  active node must be up and in the primary role.


[[s-three-node-config]]
==== Configuring a stacked resource

In the following example, nodes are named 'alice', 'bob', and
'charlie', with 'alice' and 'bob' forming a two-node cluster, and
'charlie' being the backup node.

[source,drbd]
----------------------------
resource r0 {
  protocol C;
  device    /dev/drbd0;
  disk      /dev/sda6;
  meta-disk internal;

  on alice {
    address    10.0.0.1:7788;
  }

  on bob {
    address   10.0.0.2:7788;
  }
}

resource r0-U {
  protocol A;

  stacked-on-top-of r0 {
    device     /dev/drbd10;
    address    192.168.42.1:7789;
  }

  on charlie {
    device     /dev/drbd10;
    disk       /dev/hda6;
    address    192.168.42.2:7789; # Public IP of the backup node
    meta-disk  internal;
  }
}
----------------------------

As with any `drbd.conf` configuration file, this must be distributed
across all nodes in the cluster -- in this case, three nodes. Notice
the following extra keyword not found in an unstacked resource
configuration:

.`stacked-on-top-of`
This option informs DRBD that the resource which contains it is a
stacked resource. It replaces one of the `on` sections normally found
in any resource configuration. Do not use `stacked-on-top-of` in an
lower-level resource.


NOTE: It is not a requirement to use <<fp-protocol-a,Protocol A>> for
stacked resources. You may select any of DRBD's replication protocols
depending on your application.

[[f-single-stacked]]
.Single stacked setup
image::images/single-stacked.svg[]


[[s-three-node-enable]]
==== Enabling stacked resources

To enable a stacked resource, you first enable its lower-level
resource and promote it:
----------------------------
drbdadm up r0
drbdadm primary r0
----------------------------

As with unstacked resources, you must create DRBD meta data on the
stacked resources. This is done using the following command:

----------------------------
# drbdadm create-md --stacked r0-U
----------------------------

Then, you may enable the stacked resource:

----------------------------
# drbdadm up --stacked r0-U
# drbdadm primary --stacked r0-U
----------------------------

After this, you may bring up the resource on the backup node, enabling
three-node replication:

----------------------------
# drbdadm create-md r0-U
# drbdadm up r0-U
----------------------------

In order to automate stacked resource management, you may integrate
stacked resources in your cluster manager configuration.
ifdef::drbd-only[]
See <<s-pacemaker-stacked-resources>> for information on doing this in a
cluster managed by the Pacemaker cluster management framework.
endif::drbd-only[]

[[s-permanently-diskless-nodes]]
=== Permanently diskless nodes

A node might be permanently diskless in DRBD. Here is a configuration example
showing a resource with 3 diskful nodes (servers) and one permanently diskless
node (client).

-------------------------------------
resource kvm-mail {
  device      /dev/drbd6;
  disk        /dev/vg/kvm-mail;
  meta-disk   internal;

  on store1 {
    address   10.1.10.1:7006;
    node-id   0;
  }
  on store2 {
    address   10.1.10.2:7006;
    node-id   1;
  }
  on store3 {
    address   10.1.10.3:7006;
    node-id   2;
  }

  on for-later-rebalancing {
    address   10.1.10.4:7006;
    node-id   3;
  }

  # DRBD "client"
  floating 10.1.11.6:8006 {
    disk      none;
    node-id   4;
  }

  # rest omitted for brevity
  ...
}
-------------------------------------

For permanently diskless nodes no bitmap slot gets allocated. For such nodes the
diskless status is displayed in green color since it is not an error or unexpected
state. See <<s-drbd-client-internals>> for internal details.

[[s-rebalance-workflow]]
=== Data rebalancing

indexterm:[rebalance]Given the (example) policy that data needs to be available on 3 nodes,
you need at least 3 servers for your setup.

Now, as your storage demands grow, you will encounter the need for
additional servers. Rather than having to buy 3 more servers at the same
time, you can _rebalance_ your data across a single additional node.


.DRBD data rebalancing
image::images/rebalance.svg[]

In the figure above you can see the _before_ and _after_ states: from
3 nodes with three 25TiB volumes each (for a net 75TiB), to 4 nodes,
with net 100TiB.

To redistribute the data across your cluster you have to choose a _new_ node,
and one where you want to remove this DRBD resource. +
Please note that removing the resource from a currently _active_ node (i.e.
where DRBD is _Primary_) will involve either migrating the service or running
this resource on this node as a <<s-drbd-client,DRBD client>>; it's easier to
choose a node in _Secondary_ role. (Of course, that might not always be possible.)


[[s-prepare_a_bitmap_slot]]
==== Prepare a bitmap slot

You will need to have a free <<s-quick-sync-bitmap,bitmap slot>> for
temporary use, on each of the nodes that have the resource that is to be moved.

You can allocate one more at
<<s-first-time-up,`drbdadm create-md` time>>, or simply put a placeholder in
your configuration, so that `drbdadm` sees that it should reserve one more slot:

---------
resource r0 {
  ...
  on for-later-rebalancing {
    address   10.254.254.254:65533;
    node-id   3;
  }
}
---------

[NOTE]
=================
If you need to make that slot available during live use,
you will have to

. dump the metadata,
. enlarge the metadata space,
. edit the dumpfile,
. load the changed metadata.

In a future version `drbdadm` will have a shortcut for you; most
probably you'll be able to say `drbdadm resize --peers _N_` and have the
kernel rewrite the metadata for you.
=================


[[s-preparing_and_activating_the_new_node]]
==== Preparing and activating the new node

First of all you have to
create the underlying storage volume on the new node (using e.g. `lvcreate`).
Then the placeholder in the configuration can be filled with the correct host
name, address, and storage path. Now copy the resource configuration to all
relevant nodes.


On the new node initialize the meta-data (once) by doing

-------------
# drbdadm create-md <resource>
v09 Magic number not found
Writing meta data...
initialising activity log
NOT initializing bitmap
New drbd meta data block successfully created.
-------------


[[s-starting_the_initial_sync]]
==== Starting the initial sync

Now the new node needs to get the data.

This is done by defining the network connection on the existing nodes via

-------------------------------------
# drbdadm adjust <resource>
-------------------------------------

and starting the DRBD device on the new node via

-------------------------------------
# drbdadm up <resource>
-------------------------------------


[[s-check_connectivity]]
==== Check connectivity

At this time please do a
------------
# drbdadm status <resource>
------------

on the new node, and check that _all_ other nodes are connected.

[[s-after_the_initial_sync]]
==== After the initial sync

As soon as the new host is _UpToDate_, one of the
other nodes in the configuration can be renamed to `for-later-rebalancing`, and
kept for another migration.

NOTE: Perhaps you want to comment the section; although that has the risk that
doing a `drbdadm create-md` for a new node has too few bitmap slots for the
next rebalancing. +
It might be easier to use a reserved (unused) IP address and host name.

Copy the changed configuration around again, and use it by running

-------------------------------------
# drbdadm adjust <resource>
-------------------------------------

on all nodes.


[[s-cleaning_up]]
==== Cleaning up

On the one node that had the data up to now, but isn't used anymore for this
resource, you can now take the DRBD device down by starting

-------------------
# drbdadm down <resource>
-------------------

Now the lower level storage device isn't used anymore, and can either be
re-used for other purposes or, if it is a logical volume, its space can be
returned to the volume group via `lvremove`.

[[s-conclusion_and_further_steps]]
==== Conclusion and further steps

One of the resources has been migrated to the new node. The same could be done
for one or more other resources, to free space on two or three nodes in the
existing cluster.

Then new resources can be configured, as there are enough nodes with free space
to achieve 3-way redundancy again.

[[s-configuring-quorum]]
=== Configuring quorum

In order to avoid split brain or diverging data of replicas one has to configure
fencing. All the options for fencing rely on redundant communication in the end.
That might be in the form of a management network that connects the nodes to the IPMI
network interfaces of the peer machines. In case of the crm-fence-peer script it
is necessary that Pacemakers communication stays available when DRBD's network
link breaks.

The quorum mechanism on the other hand takes a completely difference approach. The basic
idea is that a cluster partition may only modify the replicated data set if the
number of nodes that can communicate is greater than the half of the overall number
of nodes. A node of such a partition _has quorum_.
On the other hand a node does not have quorum needs to guarantee that the replicated
data set it not touched, that it does not create a diverging data set.

The quorum implementation in DRBD gets enabled by setting the `quorum` resource
option to `majority`, `all` or a numeric value. Where `majority` selects the
behavior that was described in the previous paragraph.

[[s-guaranteed_minimal_redundancy]]
==== Guaranteed minimal redundancy

By default every node with a disk gets a vote in the quorum election. In other words
only diskless nodes do not count. So a partition with two _Inconsistent_ disks gets
quorum, while a partition with one _UpToDate_ node will have quorum in a 3 node
cluster. By configuring `quorum-minimum-redundancy` this behavior can be changed so that
only nodes that are `UpToDate` have a vote in the quorum election. The option takes the
same arguments as the `quorum` option.

With this option you express that you rather want to wait until eventually necessary
resync operations finish before any services start. So in a way you prefer
that the minimal redundancy of your data is guaranteed over the availability of your
service. Financial data and services is an example that comes to mind.

Consider following example for a 5 node cluster. It requires a partitions to have at
least 3 nodes, and two of them must be `UpToDate`:

-------------------------------------
resource quorum-demo {
  options {
    quorum majority;
    quorum-minimum-redundancy 2;
    ...
  }
}
-------------------------------------

[[s-actions_on_loss_of_quorum]]
==== Actions on loss of quorum

When a node that is running the service loses quorum it needs to cease write-operations
on the data set immediately. That means that IO immediately starts to complete all IO
requests with errors. Usually that means that a graceful shutdown is not possible, since
that would require more modifications to the data set. The IO errors propagate from
the block level to the file system and from the file system to the user space application(s).

Ideally the application simply terminates in case of IO errors. This allows then Pacemaker
to unmount the filesystem and to demote the DRBD resource to secondary role.
If that is true you should set the `on-no-quorum` resource option to `io-error`.
Here is an example:

-------------------------------------
resource quorum-demo {
  options {
    quorum majority;
    on-no-quorum io-error;
    ...
  }
}
-------------------------------------

If you application does not terminate on the first IO error, you can choose to freeze IO
instead and to reboot the node. Here is a configuration example:

-------------------------------------
resource quorum-demo {
  options {
    quorum majority;
    on-no-quorum suspend-io;
    ...
  }

  handlers {
    quorum-lost "echo b > /proc/sysrq-trigger";
  }
  ...
}
-------------------------------------

[[s-configuring-quorum-tiebreaker]]
==== Using a diskless node as a tiebreaker

A diskless node with connections to all nodes in a cluster can be utilized to break ties
in the quorum negotiation process.

Consider the following two-node cluster, where node A is the primary and node B is a secondary:

image::images/quorum-tiebreaker-without.svg[]

As soon as the connection between the two nodes is interrupted, they lose quorum
and the application on top of the cluster cannot write data anymore.

image::images/quorum-tiebreaker-without-disconnect.svg[]

Now if we add a third node, C, to the cluster and configure it as diskless, we
can take advantage of the tiebreaker mechanism.

image::images/quorum-tiebreaker.svg[]

In this case, when the primary and secondary nodes lose connection, they can
still "see" the diskless tiebreaker. Because of this, the primary can continue
working, while the secondary demotes its disk to Outdated and the service can
therefore not be migrated there.

image::images/quorum-tiebreaker-disconnect.svg[]

There are a few special cases in case two connections fail. Consider the following
scenario:

image::images/quorum-tiebreaker-disconnect-case2.svg[]

In this case, the tiebreaker node forms a partition with the primary node. The primary
therefore keeps quorum, while the secondary becomes outdated. Note that the secondary's
disk state will be "UpToDate", but regardless it cannot be promoted to primary because
it lacks quorum.

Let's consider the possibility of the primary losing connection to the tiebreaker:

image::images/quorum-tiebreaker-disconnect-case3.svg[]

In this case, the primary becomes unusable and goes into a "quorum suspended" state. This
effectively results in the application on top of DRBD receiving I/O errors. A cluster manager
could then promote node B to primary and keep the service running there instead.

In case the diskless tiebreaker "switches sides", we need to avoid data divergence. Consider
this scenario:

image::images/quorum-tiebreaker-disconnect-case1a.svg[]

The connection between the primary and secondary has failed, and the application is continuing
to run on the primary, when the primary suddenly loses its connection to the diskless node.

In this case, no node can be promoted to primary and the cluster cannot continue to operate.

[NOTE]
====================
Protecting from data divergence always takes priority over ensuring service availability.
====================

Let's look at another scenario:

image::images/quorum-tiebreaker-disconnect-case2a.svg[]

Here, the application is running on the primary, while the secondary is unavailable. Then,
the tiebreaker first loses connection to the primary, and then reconnects to the secondary.
It is important to note here that *a node that has lost quorum cannot regain quorum by connecting
to a diskless node*. Thus, in this case, no node has quorum and the cluster halts.

[[s-quorum_last_man_standing]]
==== Last man standing

It needs to be mentioned that nodes that leave a cluster gracefully are counted
different than failed nodes. In this context _leaving gracefully_ means that the leaving
node marked it's data as Outdated, and that it was able to tell the remaining nodes
that it's data is Outdated.

In a group of nodes where all disks are Outdated no one in that group can promote to Primary
footnote:[The exception is a manual promotion with the --force flag. We assume that
one that uses the --force knows what he is doing.].

An implication is that, if a cluster is left by all but one node, it can keep quorum as
long as all the others left in a graceful way. If a single node leaves ungraceful,
it has to assume that all away nodes form a partition and have access to up-to-date
data. If the other partition might (potentially) be larger than my own partition it
looses quorum.

[[s-removing-DRBD]]
=== Removing DRBD

For the unlikely case that you want to remove DRBD, here are the necessary steps.

. Stop the services and unmount the filesystems on top of the DRBD volumes. In case you are
  using a cluster manager, make sure it ceases to control the services fist.
. Stop the DRBD resource(s) by using `drbdadm down <res>` or `drbdadm down all`
.. In case the DRBD resource was using internal meta-data you might choose to
   resize the filesystem to cover all of the backing device's space. This step
   effectively removes DRBD's meta-data. This is an action that can not be reversed
   easily. You can do that with `resize2fs <backing_dev>` for ext[234] family of file systems.
   It supports resizing of unmounted filesystem and under certain conditions also online
   grow. XFS can be grown online only with the `xfs_growfs` command.
. Mount the backing device(s) directly, start the services on top of them
. Unload the drbd kernel driver modules with `rmmod drbd_transport_tcp` and `rmmod drbd`.
. Uninstall the drbd software packages.

//- list divider. Keep it otherwise next included section is interpreted as list element
