<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
  <!ENTITY drbdsetup_options SYSTEM "./drbdsetup-options.xml">
]>

<refentry id="re-drbdconf">
  &drbdsetup_options;

  <refentryinfo>
    <date>17 January 2018</date>

    <productname>DRBD</productname>

    <productnumber>9.0.x</productnumber>
  </refentryinfo>

  <refmeta>
    <refentrytitle>drbd.conf</refentrytitle>

    <manvolnum>5</manvolnum>

    <refmiscinfo class="manual">Configuration Files</refmiscinfo>
  </refmeta>

  <refnamediv>
    <refname>drbd.conf</refname>

    <refpurpose>DRBD Configuration Files<indexterm significance="normal">
        <primary>drbd.conf</primary>
      </indexterm></refpurpose>
  </refnamediv>

  <refsect1>
    <title>Introduction</title>

    <para>DRBD implements block devices which replicate their data to all nodes
    of a cluster.  The actual data and associated metadata are usually stored
    redundantly on "ordinary" block devices on each cluster node.</para>

    <para>Replicated block devices are called
    <option>/dev/drbd<replaceable>minor</replaceable></option> by default. They
    are grouped into resources, with one or more devices per resource.
    Replication among the devices in a resource takes place in chronological order. With DRBD,
    we refer to the devices inside a resource as <emphasis>volumes</emphasis>.</para>

    <para>In DRBD 9, a resource can be replicated between two or more cluster
    nodes.  The connections between cluster nodes are point-to-point links, and
    use TCP or a TCP-like protocol.  All nodes must be directly
    connected.</para>

    <para>DRBD consists of low-level user-space components
    which interact with the kernel and perform basic operations
    (<option>drbdsetup</option>, <option>drbdmeta</option>), a high-level
    user-space component which understands and processes the DRBD configuration
    and translates it into basic operations of the low-level components
    (<option>drbdadm</option>), and a kernel component.</para>

    <para>The default DRBD configuration consists of
    <option>/etc/drbd.conf</option> and of additional files included from
    there, usually <option>global_common.conf</option> and all
    <option><replaceable>*</replaceable>.res</option> files inside
    <option>/etc/drbd.d/</option>.  It has turned out to be useful to define
    each resource in a separate
    <option><replaceable>*</replaceable>.res</option> file.</para>

    <para>The configuration files are designed so that each cluster node can
    contain an identical copy of the entire cluster configuration. The host
    name of each node determines which parts of the configuration apply
    (<option>uname -n</option>).
    It is highly recommended to keep the cluster configuration on all
    nodes in sync by manually copying it to all nodes, or by automating the
    process with <option>csync2</option> or a similar tool.</para>
  </refsect1>

  <refsect1>
    <title>Example Configuration File</title>
    <para>
    <programlisting format="linespecific">
global {
	usage-count yes;
	udev-always-use-vnr;
}
resource r0 {
      net {
	      cram-hmac-alg sha1;
	      shared-secret "FooFunFactory";
      }
      volume 0 {
	      device    "/dev/drbd1";
	      disk      "/dev/sda7";
	      meta-disk internal;
      }
      on "alice" {
	      node-id   0;
	      address   10.1.1.31:7000;
      }
      on "bob" {
	      node-id   1;
	      address   10.1.1.32:7000;
      }
      connection {
	      host      "alice"  port 7000;
	      host      "bob"    port 7000;
	      net {
			protocol C;
	      }
      }
}</programlisting>
    This example defines a resource <option>r0</option> which contains a single replicated
    device with volume number 0.  The resource is replicated among hosts <option>alice</option>
    and <option>bob</option>, which have the IPv4 addresses <option>10.1.1.31</option> and <option>10.1.1.32</option> and the
    node identifiers 0 and 1, respectively.  On both hosts, the replicated
    device is called <option>/dev/drbd1</option>, and the actual data and metadata are stored on
    the lower-level device <option>/dev/sda7</option>.  The connection between the hosts uses
    protocol C.</para>

    <para>Enclose strings within double-quotation marks (") to differentiate them from resource keywords. Please refer to the <ulink
	url="http://www.drbd.org/users-guide/"><citetitle>DRBD User's Guide</citetitle></ulink>
      for more examples.</para>

  </refsect1>

  <refsect1>
    <title>File Format</title>

    <para>DRBD configuration files consist of sections, which contain other
    sections and parameters depending on the section types. Each section
    consists of one or more keywords, sometimes a section name, an opening
    brace (<quote>{</quote>), the section's contents, and a closing brace
    (<quote>}</quote>). Parameters inside a section consist of a keyword,
    followed by one or more keywords or values, and a semicolon
    (<quote>;</quote>).</para>

    <para>Some parameter values have a default scale which applies when a plain
    number is specified (for example Kilo, or 1024 times the numeric value).
    Such default scales can be overridden by using a suffix (for example,
    <option>M</option> for Mega).  The common suffixes <option>K</option> =
    2^10 = 1024, <option>M</option> = 1024 K, and <option>G</option> = 1024 M
    are supported.</para>

    <para>Comments start with a hash sign (<quote>#</quote>) and extend to the
    end of the line. In addition, any section can be prefixed with the keyword
    <option>skip</option>, which causes the section and any sub-sections to be
    ignored.</para>

    <para>Additional files can be included with the <option>include
    <replaceable>file-pattern</replaceable></option> statement (see
    <citerefentry><refentrytitle>glob</refentrytitle><manvolnum>7</manvolnum></citerefentry>
    for the expressions supported in <replaceable>file-pattern</replaceable>).
    Include statements are only allowed outside of sections.</para>

    <para>The following sections are defined (indentation indicates in
    which context):
    <programlisting>
common
   [disk]
   [handlers]
   [net]
   [options]
   [startup]
global
[require-drbd-module-version-{eq,ne,gt,ge,lt,le}]
resource
   connection
      multiple path | 2 host
      [net]
      [volume]
         [peer-device-options]
      [peer-device-options]
   connection-mesh
      [net]
   [disk]
   floating
   handlers
   [net]
   on
      volume
         disk
         [disk]
   options
   stacked-on-top-of
   startup</programlisting>
    Sections in brackets affect other parts of the configuration: inside the
    <option>common</option> section, they apply to all resources. A
    <option>disk</option> section inside a <option>resource</option> or
    <option>on</option> section applies to all volumes of that resource, and a
    <option>net</option> section inside a <option>resource</option> section
    applies to all connections of that resource. This allows to avoid repeating
    identical options for each resource, connection, or volume. Options can
    be overridden in a more specific <option>resource</option>,
    <option>connection</option>, <option>on</option>, or
    <option>volume</option> section.</para>

    <para><option>peer-device-options</option> are
    <option>resync-rate</option>, <option>c-plan-ahead</option>, <option>c-delay-target</option>,
    <option>c-fill-target</option>, <option>c-max-rate</option> and <option>c-min-rate</option>.
    Due to backward comapatibility they can be specified in any disk options section as well.
    They are inherited into all relevant connections. If they are given on <option>connection</option>
    level they are inherited to all volumes on that connection. A <option>peer-device-options</option>
    section is started with the <option>disk</option> keyword.</para>

    <refsect2>
      <title>Sections</title>

      <variablelist>
        <varlistentry>
          <term><option>common</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>common</secondary>
              </indexterm> This section can contain each a
              <option>disk</option>, <option>handlers</option>,
              <option>net</option>, <option>options</option>, and
              <option>startup</option> section. All resources inherit the
              parameters in these sections as their default values.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>connection</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>connection</secondary>
              </indexterm> Define a connection between two hosts. This section
              must contain two <option>host</option> parameters or multiple
	      <option>path sections</option>.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>path</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>path</secondary>
              </indexterm> Define a path between two hosts. This section
              must contain two <option>host</option> parameters.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>connection-mesh</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>connection-mesh</secondary>
              </indexterm> Define a connection mesh between multiple hosts. This section
              must contain a <option>hosts</option> parameter, which has the
	      host names as arguments. This section is a shortcut to define many
	      connections which share the same network options.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>disk</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>disk</secondary>
              </indexterm> Define parameters for a volume. All parameters in
              this section are optional.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>floating <replaceable><optional>address-family</optional></replaceable> <replaceable>addr</replaceable>:<replaceable>port</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>floating</secondary>
              </indexterm> Like the <option>on</option> section, except that
              instead of the host name a network address is used to
              determine if it matches a <option>floating</option>
              section.</para>

	    <para>The <option>node-id</option> parameter in this section is
	      required.  If the <option>address</option> parameter is not
	      provided, no connections to peers will be created by default.
	      The <option>device</option>, <option>disk</option>, and
	      <option>meta-disk</option> parameters must be defined in, or
	      inherited by, this section.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>global</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>global</secondary>
              </indexterm> Define some global parameters.  All parameters in
              this section are optional.  Only one <option>global</option>
              section is allowed in the configuration.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>require-drbd-module-version-{eq,ne,gt,ge,lt,le}</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>require-drbd-module-version-{eq,ne,gt,ge,lt,le}</secondary>
              </indexterm> This statement contains one of the valid forms and a three digit version number
				  (e.g., <option>require-drbd-module-version-eq&#160;9.0.16;</option>). If the currently loaded DRBD
				  kernel module does not match the specification, parsing is aborted. Comparison operator names have
              same semantic as in <citerefentry><refentrytitle>test</refentrytitle><manvolnum>1</manvolnum></citerefentry>.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>handlers</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>handlers</secondary>
              </indexterm> Define handlers to be invoked when certain events
              occur. The kernel passes the resource name in the first
              command-line argument and sets the following environment
              variables depending on the event's context:
              <itemizedlist>
                <listitem>
                        <para>For events related to a particular device:
                        the device's minor number in
                        <option>DRBD_MINOR</option>, the device's volume number
                        in <option>DRBD_VOLUME</option>.
                  </para>
                </listitem>
                <listitem>
                  <para>For events related to a particular device on a
                  particular peer: the connection endpoints in
                  <option>DRBD_MY_ADDRESS</option>,
                  <option>DRBD_MY_AF</option>,
                  <option>DRBD_PEER_ADDRESS</option>, and
                  <option>DRBD_PEER_AF</option>; the device's local minor
                  number in <option>DRBD_MINOR</option>, and the device's volume
                  number in <option>DRBD_VOLUME</option>.
                  </para>
                </listitem>
                <listitem>
                  <para>For events related to a particular connection: the
                  connection endpoints in <option>DRBD_MY_ADDRESS</option>,
                  <option>DRBD_MY_AF</option>,
                  <option>DRBD_PEER_ADDRESS</option>, and
                  <option>DRBD_PEER_AF</option>; and, for each device defined for
                  that connection: the device's minor number in
                  <option>DRBD_MINOR_<replaceable>volume-number</replaceable></option>.
                  </para>
                </listitem>
                <listitem>
                  <para>For events that identify a device, if a lower-level
                  device is attached, the lower-level device's device name is
                  passed in <option>DRBD_BACKING_DEV</option> (or
                  <option>DRBD_BACKING_DEV_<replaceable>volume-number</replaceable></option>).
                  </para>
              </listitem>
              </itemizedlist>
            </para>
            <para>All parameters in this section are optional. Only a single
            handler can be defined for each event; if no handler is defined,
            nothing will happen.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><anchor xml:id="net"/><option>net</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>net</secondary>
              </indexterm> Define parameters for a connection. All parameters
              in this section are optional.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>on</option> <option><replaceable>host-name</replaceable></option> <replaceable><optional>...</optional></replaceable></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>on</secondary>
              </indexterm> Define the properties of a resource on a particular
              host or set of hosts. Specifying more than one host name can
              make sense in a setup with IP address failover, for
              example. The <replaceable>host-name</replaceable> argument must
              match the Linux host name (<option>uname -n</option>).</para>

            <para>Usually contains or inherits at least one
            <option>volume</option> section. The <option>node-id</option> and
            <option>address</option> parameters must be defined in this
            section. The <option>device</option>, <option>disk</option>, and
            <option>meta-disk</option> parameters must be defined in, or
            inherited by, this section.</para>

            <para>A normal configuration file contains two or more
            <option>on</option> sections for each resource. Also see the
            <option>floating</option> section.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>options</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>options</secondary>
              </indexterm> Define parameters for a resource. All parameters
              in this section are optional.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>resource</option> <option><replaceable>name</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>resource</secondary>
              </indexterm> Define a resource. Usually contains at least two
              <option>on</option> sections and at least one
              <option>connection</option> section. <!-- FIXME? auto-mesh? --></para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>stacked-on-top-of <replaceable>resource</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>stacked-on-top-of</secondary>
            </indexterm> Used instead of an <option>on</option> section for
            configuring a stacked resource with three to four nodes.</para>
	  <para>Starting with DRBD 9, stacking is deprecated.  It is advised
	    to use resources which are replicated among more than two nodes
	    instead.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>startup</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>startup</secondary>
              </indexterm> The parameters in this section determine the
              behavior of a resource at startup time.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>volume</option> <option><replaceable>volume-number</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>volume</secondary>
              </indexterm> Define a volume within a resource. The volume
              numbers in the various <option>volume</option> sections of a
              resource define which devices on which hosts form a replicated
              device.</para>
          </listitem>
        </varlistentry>
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>connection</option> Parameters</title>

      <variablelist>
        <varlistentry>
          <term><option>host <replaceable>name</replaceable></option>
            <optional><option>address
                <optional>address-family</optional>
                <replaceable>address</replaceable></option></optional>
            <optional><option>port
                <replaceable>port-number</replaceable></option></optional></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>host</secondary>
              </indexterm>
            Defines an endpoint for a connection. Each <option>host</option>
            statement refers to an <option>on</option> section in a resource.
            If a port number is defined, this endpoint will use the specified
            port instead of the port defined in the <option>on</option>
            section.  Each <option>connection</option> section must contain
            exactly two <option>host</option> parameters. Instead of two
	    <option>host</option> parameters the connection may contain
	    multiple <option>path</option> sections.</para>
          </listitem>
        </varlistentry>
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>path</option> Parameters</title>

      <variablelist>
        <varlistentry>
          <term><option>host <replaceable>name</replaceable></option>
            <optional><option>address
                <optional>address-family</optional>
                <replaceable>address</replaceable></option></optional>
            <optional><option>port
                <replaceable>port-number</replaceable></option></optional></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>host</secondary>
              </indexterm>
            Defines an endpoint for a connection. Each <option>host</option>
            statement refers to an <option>on</option> section in a resource.
            If a port number is defined, this endpoint will use the specified
            port instead of the port defined in the <option>on</option>
            section.  Each <option>path</option> section must contain
            exactly two <option>host</option> parameters.</para>
          </listitem>
        </varlistentry>
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>connection-mesh</option> Parameters</title>

      <variablelist>
        <varlistentry>
          <term><option>hosts <arg choice="plain" rep="repeat"><replaceable>name</replaceable></arg></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>host</secondary>
              </indexterm>
            Defines all nodes of a mesh. Each
	    <option><replaceable>name</replaceable></option> refers to an
	    <option>on</option> section in a resource. The port
	    that is defined in the <option>on</option> section will
	    be used.</para>
          </listitem>
        </varlistentry>
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>disk</option> Parameters</title>

      <variablelist>
	<pick_drbdsetup_option name="al-extents"/>
	<pick_drbdsetup_option name="al-updates"/>
	<pick_drbdsetup_option name="disk-barrier"/>
	<pick_drbdsetup_option name="disk-timeout"/>
	<pick_drbdsetup_option name="md-flushes"/>
	<pick_drbdsetup_option name="on-io-error"/>
	<pick_drbdsetup_option name="read-balancing"/>
	<pick_drbdsetup_option name="resync-after"/>
	<pick_drbdsetup_option name="rs-discard-granularity"/>
	<pick_drbdsetup_option name="discard-zeroes-if-aligned"/>
	<pick_drbdsetup_option name="disable-write-same"/>
	<!-- <pick_drbdsetup_option name="size"/> -->
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>peer-device-options</option> Parameters</title>
      <para>Please note that you open the section with the <option>disk</option> keyword.</para>
      <variablelist>
	<pick_drbdsetup_option name="c-delay-target"/>
	<pick_drbdsetup_option name="c-min-rate"/>
	<pick_drbdsetup_option name="resync-rate"/>
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>global</option> Parameters</title>

      <variablelist>
        <pick_drbdsetup_option name="dialog-refresh"/>
        <pick_drbdsetup_option name="disable-ip-verification"/>
        <pick_drbdsetup_option name="usage-count"/>
        <pick_drbdsetup_option name="udev-always-use-vnr"/>
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>handlers</option> Parameters</title>

      <variablelist>
        <varlistentry>
          <term><option>after-resync-target <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>after-resync-target</secondary>
              </indexterm> Called on a resync target when a node state
              changes from <option>Inconsistent</option> to <option>Consistent</option> when a resync finishes.
              This handler can be used for removing the snapshot created in the
              <option>before-resync-target</option> handler.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>before-resync-target <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>before-resync-target</secondary>
              </indexterm> Called on a resync target before a resync begins.
              This handler can be used for creating a snapshot of the
              lower-level device for the duration of the resync: if the resync
              source becomes unavailable during a resync, reverting to the
              snapshot can restore a consistent state.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>before-resync-source <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>before-resync-source</secondary>
              </indexterm> Called on a resync source before a resync begins.
              </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>out-of-sync <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>out-of-sync</secondary>
              </indexterm> Called on all nodes after a <option>verify</option> finishes and out-of-sync blocks
              were found. This handler is mainly used for monitoring purposes.
              An example would be to call a script that sends an alert SMS.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>quorum-lost <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>quorum-lost</secondary>
              </indexterm> Called on a Primary that lost quorum.
              This handler is usually used to reboot the node if it is not possible
              to restart the application that uses the storage on top of DRBD.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>fence-peer <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>fence-peer</secondary>
              </indexterm> Called when a node should fence a resource on a
              particular peer. The handler should not use the same
              communication path that DRBD uses for talking to the peer.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>unfence-peer <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>unfence-peer</secondary>
              </indexterm> Called when a node should remove fencing constraints
	      from other nodes.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>initial-split-brain <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>initial-split-brain</secondary>
              </indexterm> Called when DRBD connects to a peer and detects that
              the peer is in a split-brain state with the local node.  This
              handler is also called for split-brain scenarios which will be
              resolved automatically.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>local-io-error <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>local-io-error</secondary>
              </indexterm> Called when an I/O error occurs on a lower-level device.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>pri-lost <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>pri-lost</secondary>
              </indexterm>
              The local node is currently primary, but DRBD believes that it
              should become a sync target. The node should give up its primary
              role.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>pri-lost-after-sb <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>pri-lost-after-sb</secondary>
              </indexterm> The local node is currently primary, but it has lost
              the after-split-brain auto recovery procedure. The node should be
              abandoned. <!-- FIXME: What is "abandon a node" supposed to mean? --></para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>pri-on-incon-degr <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>pri-on-incon-degr</secondary>
	      </indexterm> The local node is primary, and neither the local
	    lower-level device nor a lower-level device on a peer is up to
	    date.  (The primary has no device to read from or to write
	    to.)</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>split-brain <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>split-brain</secondary>
              </indexterm> DRBD has detected a split-brain situation which
              could not be resolved automatically.  Manual recovery is
              necessary. This handler can be used to call for administrator
              attention.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>disconnected <replaceable>cmd</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>disconnected</secondary>
              </indexterm> A connection to a peer went down. The handler can
	      learn about the reason for the disconnect from the <option>DRBD_CSTATE</option>
	      environment variable.</para>
          </listitem>
        </varlistentry>

      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>net</option> Parameters</title>

      <variablelist>
        <pick_drbdsetup_option name="after-sb-0pri"/>
        <pick_drbdsetup_option name="after-sb-1pri"/>
        <pick_drbdsetup_option name="after-sb-2pri"/>
        <pick_drbdsetup_option name="allow-two-primaries"/>
        <pick_drbdsetup_option name="always-asbp"/>
        <pick_drbdsetup_option name="connect-int"/>
        <pick_drbdsetup_option name="cram-hmac-alg"/>
        <pick_drbdsetup_option name="csums-alg"/>
        <pick_drbdsetup_option name="csums-after-crash-only"/>
        <pick_drbdsetup_option name="data-integrity-alg"/>
        <pick_drbdsetup_option name="fencing"/>
        <pick_drbdsetup_option name="ko-count"/>
        <pick_drbdsetup_option name="max-buffers"/>
        <pick_drbdsetup_option name="max-epoch-size"/>
        <pick_drbdsetup_option name="on-congestion"/>
        <pick_drbdsetup_option name="ping-int"/>
        <pick_drbdsetup_option name="ping-timeout"/>
        <pick_drbdsetup_option name="socket-check-timeout"/>
        <pick_drbdsetup_option name="protocol"/>
        <pick_drbdsetup_option name="rcvbuf-size"/>
        <pick_drbdsetup_option name="rr-conflict"/>
        <pick_drbdsetup_option name="shared-secret"/>
        <pick_drbdsetup_option name="sndbuf-size"/>
        <pick_drbdsetup_option name="tcp-cork"/>
        <pick_drbdsetup_option name="timeout"/>
        <pick_drbdsetup_option name="transport"/>
        <pick_drbdsetup_option name="use-rle"/>
        <pick_drbdsetup_option name="verify-alg"/>
        <pick_drbdsetup_option name="allow-remote-read"/>
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>on</option> Parameters</title>

      <variablelist>
        <varlistentry>
          <term><option>address
              <replaceable><optional>address-family</optional></replaceable>
              <replaceable>address</replaceable>:<replaceable>port</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>address</secondary>
              </indexterm> Defines the address family, address, and port of a
              connection endpoint.</para>

	    <para>The address families <option>ipv4</option>,
	      <option>ipv6</option>, <option>ssocks</option> (Dolphin
	      Interconnect Solutions' "super sockets"), <option>sdp</option>
	      (Infiniband Sockets Direct Protocol), and <option>sci</option>
	      are supported (<option>sci</option> is an alias for
	      <option>ssocks</option>).  If no address family is specified,
	      <option>ipv4</option> is assumed.  For all address families
	      except <option>ipv6</option>, the address is specified in IPV4
	      address notation (for example, 1.2.3.4).  For
	      <option>ipv6</option>, the address is enclosed in brackets and
	      uses IPv6 address notation (for example, <literal
		moreinfo="none">[fd01:2345:6789:abcd::1]</literal>).  The port
	      is always specified as a decimal number from 1 to 65535.</para>

            <para>On each host, the port numbers must be unique for each
              address; ports cannot be shared.</para>
          <!-- FIXME: The port should be made optional to avoid duplicate
                      information here and in the connection sections. -->
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>node-id <replaceable>value</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>node-id</secondary>
              </indexterm>
              Defines the unique node identifier for a node in the cluster.
              Node identifiers are used to identify individual nodes in the
              network protocol, and to assign bitmap slots to nodes in the
              metadata.</para>

            <para>Node identifiers can only be reasssigned in a cluster when
              the cluster is down.  It is essential that the node identifiers
              in the configuration and in the device metadata are changed
              consistently on all hosts.  To change the metadata, dump the
              current state with <command>drbdmeta dump-md</command>, adjust the bitmap slot
              assignment, and update the metadata with
              <command>drbdmeta restore-md</command>.</para>

            <para>The <option>node-id</option> parameter exists since DRBD 9.
              Its value ranges from 0 to 16; there is no default.</para>
          </listitem>
        </varlistentry>
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>options</option> Parameters (Resource Options)</title>

      <variablelist>
        <pick_drbdsetup_option name="auto-promote"/>
        <pick_drbdsetup_option name="cpu-mask"/>
        <pick_drbdsetup_option name="on-no-data-accessible"/>
        <pick_drbdsetup_option name="peer-ack-window"/>
        <pick_drbdsetup_option name="peer-ack-delay"/>
        <pick_drbdsetup_option name="quorum"/>
	<pick_drbdsetup_option name="quorum-minimum-redundancy"/>
        <pick_drbdsetup_option name="on-no-quorum"/>
        <pick_drbdsetup_option name="on-suspended-primary-outdated"/>
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>startup</option> Parameters</title>

      <para>The parameters in this section define the behavior of DRBD at
        system startup time, in the DRBD init script.  They have no effect once
        the system is up and running.</para>

      <variablelist>
	<pick_drbdsetup_option name="degr-wfc-timeout"/>
	<pick_drbdsetup_option name="outdated-wfc-timeout"/>

        <varlistentry>
          <term><option>stacked-timeouts</option></term>

          <listitem>
            <para>On stacked devices, the <option>wfc-timeout</option> and
              <option>degr-wfc-timeout</option> parameters in the configuration
              are usually ignored, and both timeouts are set to twice the
              <option>connect-int</option> timeout.  The
              <option>stacked-timeouts</option> parameter tells DRBD to use the
              <option>wfc-timeout</option> and
              <option>degr-wfc-timeout</option> parameters as defined in the
              configuration, even on stacked devices.  Only use this parameter
              if the peer of the stacked resource is usually not available, or
              will not become primary.  Incorrect use of this parameter can
              lead to unexpected split-brain scenarios.</para>
          </listitem>
        </varlistentry>

	<pick_drbdsetup_option name="wait-after-sb"/>
	<pick_drbdsetup_option name="wfc-timeout"/>
      </variablelist>
    </refsect2>

    <refsect2>
      <title>Section <option>volume</option> Parameters</title>

      <variablelist>
        <varlistentry>
          <term><option>device /dev/drbd<replaceable>minor-number</replaceable></option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>device</secondary>
              </indexterm> Define the device name and minor number of a
              replicated block device.  This is the device that applications
              are supposed to access; in most cases, the device is not used
              directly, but as a file system. This parameter is required and
              the standard device naming convention is assumed.</para>

            <para>In addition to this device, udev will create
              <option>/dev/drbd/by-res/<replaceable>resource</replaceable>/<replaceable>volume</replaceable></option>
              and <option>/dev/drbd/by-disk/<replaceable>lower-level-device</replaceable></option>
              symlinks to the device.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
	  <term><option>disk</option>
	      <group choice="req" rep="norepeat">
		<arg rep="norepeat">disk</arg>
		<arg choice="plain" rep="norepeat"><option>none</option></arg>
	      </group>
	    </term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>disk</secondary>
              </indexterm> Define the lower-level block device that DRBD will
              use for storing the actual data.  While the replicated drbd
              device is configured, the lower-level device must not be used directly.
              Even read-only access with tools like <citerefentry>
                <refentrytitle>dumpe2fs</refentrytitle>
                <manvolnum>8</manvolnum> </citerefentry> and similar is not
	      allowed. The keyword <option>none</option> specifies that no
	      lower-level block device is configured; this also overrides
	      inheritance of the lower-level device.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><option>meta-disk internal</option></term>

          <term><option>meta-disk <replaceable>device</replaceable></option></term>

          <term><option>meta-disk <replaceable>device</replaceable> [<replaceable>index</replaceable>]</option></term>

          <listitem>
            <para><indexterm significance="normal">
                <primary>drbd.conf</primary>

                <secondary>meta-disk</secondary>
              </indexterm> Define where the metadata of a replicated block
              device resides: it can be <option>internal</option>, meaning that the lower-level
              device contains both the data and the metadata, or on a separate
              device.</para>

            <para>When the <replaceable>index</replaceable> form of this
              parameter is used, multiple replicated devices can share the same
              metadata device, each using a separate index.  Each index
              occupies 128 MiB of data, which corresponds to a replicated
	      device size of at most 4 TiB with two cluster nodes.  We
	      recommend not to share metadata devices anymore, and to instead
	      use the lvm volume manager for creating metadata devices as
	      needed.</para>

            <para>When the <replaceable>index</replaceable> form of this
              parameter is not used, the size of the lower-level device
              determines the size of the metadata.  The size needed is 36 KiB +
              (size of lower-level device) / 32K * (number of nodes - 1).  If <!-- FIXME: al-striping, al-extents 65536? -->
              the metadata device is bigger than that, the extra space is not
              used.</para>
	    <para>This parameter is required if a <option>disk</option> other
	      than <option>none</option> is specified, and ignored if
	      <option>disk</option> is set to <option>none</option>.  A
	      <option>meta-disk</option> parameter without a
	      <option>disk</option> parameter is not allowed.</para>
          </listitem>
        </varlistentry>
      </variablelist>
    </refsect2>
  </refsect1>

  <refsect1 id="data-integrity">
    <title>Notes on data integrity</title>

    <para>DRBD supports two different mechanisms for data integrity checking:
      first, the <option>data-integrity-alg</option> network parameter allows
      to add a checksum to the data sent over the network.  Second, the online
      verification mechanism (<command moreinfo="none">drbdadm verify</command>
      and the <option>verify-alg</option> parameter) allows to check for
      differences in the on-disk data.</para>

    <para>Both mechanisms can produce false positives if the data is modified
      during I/O (i.e., while it is being sent over the network or written to
      disk).  This does not always indicate a problem: for example, some
      file systems and applications do modify data under I/O for certain
      operations.  Swap space can also undergo changes while under I/O.</para>

    <para>Network data integrity checking tries to identify data modification
      during I/O by verifying the checksums on the sender side after sending the
      data.  If it detects a mismatch, it logs an error.  The receiver also
      logs an error when it detects a mismatch.  Thus, an error logged only on
      the receiver side indicates an error on the network, and an error logged
      on both sides indicates data modification under I/O.</para>

    <para>The most recent example of systematic data corruption was identified
      as a bug in the TCP offloading engine and driver of a certain type of
      GBit NIC in 2007: the data corruption happened on the DMA transfer from
      core memory to the card. Because the TCP checksum were calculated on the
      card, the TCP/IP protocol checksums did not reveal this problem.</para>
  </refsect1>

  <refsect1>
    <title>Version</title>

    <simpara>This document was revised for version 9.0.0 of the DRBD distribution.</simpara>
  </refsect1>

  <refsect1>
    <title>Author</title>

    <simpara>Written by Philipp Reisner <email>philipp.reisner@linbit.com</email> and Lars
    Ellenberg <email>lars.ellenberg@linbit.com</email>.</simpara>
  </refsect1>

  <refsect1>
    <title>Reporting Bugs</title>

    <simpara>Report bugs to <email>drbd-user@lists.linbit.com</email>.</simpara>
  </refsect1>

  <refsect1>
    <title>Copyright</title>

    <simpara>Copyright 2001-2018 LINBIT Information Technologies, Philipp Reisner, Lars Ellenberg.
    This is free software; see the source for copying conditions. There is NO warranty; not even
    for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.</simpara>
  </refsect1>

  <refsect1>
    <title>See Also</title>

    <para><citerefentry>
        <refentrytitle>drbd</refentrytitle>

        <manvolnum>8</manvolnum>
      </citerefentry>, <citerefentry>
        <refentrytitle>drbdsetup</refentrytitle>

        <manvolnum>8</manvolnum>
      </citerefentry>, <citerefentry>
        <refentrytitle>drbdadm</refentrytitle>

        <manvolnum>8</manvolnum>
      </citerefentry>,
      <ulink url="http://www.drbd.org/users-guide/"><citetitle>DRBD User's Guide</citetitle></ulink>,
       <ulink url="http://www.drbd.org/"><citetitle>DRBD Web Site</citetitle></ulink></para>

  </refsect1>
</refentry>
