[[ch-latency]]

== Optimizing DRBD latency

This chapter deals with optimizing DRBD latency. It examines some
hardware considerations with regard to latency minimization, and
details tuning recommendations for that purpose.

[[s-latency-hardware]]
=== Hardware considerations

DRBD latency is affected by both the latency of the underlying I/O
subsystem (disks, controllers, and corresponding caches), and the
latency of the replication network.

.I/O subsystem latency
indexterm:[latency]For _rotating media_ the I/O subsystem latency is primarily a function of
disk rotation speed. Thus, using fast-spinning disks is a valid
approach for reducing I/O subsystem latency.

For _solid state media_ (like SSDs) the Flash storage controller is the
determining factor; the next most important thing is the amount of unused
capacity. Using DRBD's <<s-trim-discard>> will help you provide the controller
with the needed information which blocks it can recycle. That way, when a write
requests comes in, it can use a block that got cleaned ahead-of-time and
doesn't have to wait *now* until there's space availablefootnote:[On low-end
hardware you can help that a bit by reserving some space - just keep
10% to 20% of the total space unpartitioned.].

[[s-hardware-bbu]]
Likewise, the use of a indexterm:[battery-backed write cache]
indexterm:[BBU]
battery-backed write cache (BBWC) reduces write completion times, also
reducing write latency. Most reasonable storage subsystems come with
some form of battery-backed cache, and allow the administrator to
configure which portion of this cache is used for read and write
operations. The recommended approach is to disable the disk read cache
completely and use all available cache memory for the disk write
cache.

.Network latency
indexterm:[latency]Network latency is, in essence, the packet
indexterm:[round-trip-time]indexterm:[RTT]
round-trip time (RTT) between hosts. It is influenced by a number of
factors, most of which are irrelevant on the dedicated, back-to-back
network connections recommended for use as DRBD replication
links. Thus, it is sufficient to accept that a certain amount of
latency always exists in network links, which typically is on
the order of 100 to 200 microseconds (μs) packet RTT for Gigabit Ethernet.

Network latency may typically be pushed below this limit only by using
lower-latency network protocols, such as running DRBD over Dolphin
Express using Dolphin SuperSockets, or a 10GBe direct connection; these are
typically in the 50µs range. Even better is InfiniBand, which provides
even lower latency.


[[s-latency-overhead-expectations]]
=== Latency overhead expectations

As for throughput, when estimating the latency overhead associated
with DRBD, there are some important natural limitations to consider:

* DRBD latency is bound by that of the raw I/O subsystem.
* DRBD latency is bound by the available network latency.

The _sum_ of the two establishes the theoretical latency _minimum_
incurred to DRBDfootnote:[for protocol C, because the other node(s) have to
write to stable storage, too]. DRBD then adds to that latency a slight
additional latency overhead, which can be expected to be less than 1 percent.

* Consider the example of a local disk subsystem with a write latency
  of 3ms and a network link with one of 0.2ms. Then the expected DRBD
  latency would be 3.2 ms or a roughly 7-percent latency increase over
  just writing to a local disk.

NOTE: Latency may be influenced by a number of other factors,
including CPU cache misses, context switches, and others.


[[s-latency-iops]]
=== Latency vs. IOPs

indexterm:[latency]indexterm:[IOPs]_IOPs_ is the abbreviation of "__I/O
operations per second__".

Marketing typically doesn't like numbers that get smaller; press releases
aren't written with "__Latency reduced by 10µs, from 50µs to 40µs now!__" in
mind, they like "__Performance increased by 25%, from 20000 to now 25000
IOPs__" much more. Therefore IOPs were invented - to get a number that says
"higher is better".

So, in other words, IOPs are the reciprocal of latency. What you'll have to
keep in mind is that the method given in <<s-measure-latency>> gives you
the latency resp. the number of IOPs for a purely sequential, single-threaded
IO load, while most other documentation will give numbers for some highly
parallel loadfootnote:[Like in "__16 threads, IO-depth of 32__" - this means that 512
I/O-requests are being done in parallel!], because this gives much "prettier"
numbers. With that kind of trick DRBD does offer you 100000 IOPs, too!

So, please don't shy away from measuring serialized, single-threaded latency.
If you want lots of IOPS, run the `fio` utility with `threads=8` and an
`iodepth=16`, or some similar settings... But please keep in mind that these
number will not have any meaning to your setup, unless you're driving
a database with many tens or hundreds of client connections active at the same
time.


[[s-latency-tuning]]
=== Tuning recommendations

[[s-latency-tuning-cpu-mask]]
==== Setting DRBD's CPU mask

DRBD allows for setting an explicit CPU mask for its kernel
threads. This is particularly beneficial for applications which would
otherwise compete with DRBD for CPU cycles.

The CPU mask is a number in whose binary representation the least
significant bit represents the first CPU, the second-least significant
bit the second, and so forth. A set bit in the bitmask implies that
the corresponding CPU may be used by DRBD, whereas a cleared bit means
it must not. Thus, for example, a CPU mask of 1 (`00000001`) means
DRBD may use the first CPU only. A mask of 12 (`00001100`) implies
DRBD may use the third and fourth CPU.


An example CPU mask configuration for a resource may look like this:

[source,drbd]
----------------------------
resource <resource> {
  options {
    cpu-mask 2;
    ...
  }
  ...
}
----------------------------

IMPORTANT: Of course, in order to minimize CPU competition between
DRBD and the application using it, you need to configure your
application to use only those CPUs which DRBD does not use.

Some applications may provide for this via an entry in a configuration
file, just like DRBD itself. Others include an invocation of the
`taskset` command in an application init script.

[NOTE]
====================

It makes sense to keep the DRBD threads running on the same L2/L3 caches.

But: the numbering of CPUs doesn't have to correlate with the physical partitioning.
You can try the `lstopo` (or `hwloc-ls`) program for X11 or `hwloc-info -v -p`
for console output to get an overview of the topology.
====================


[[s-latency-tuning-mtu-size]]
==== Modifying the network MTU

It may be beneficial to change the replication network's
maximum transmission unit (MTU) size to a value higher than the
default of 1500 bytes. Colloquially, this is referred to as
indexterm:[Jumbo frames] "enabling Jumbo frames".

The MTU may be changed using the following commands:
----------------------------
# ifconfig <interface> mtu <size>
----------------------------
or
----------------------------
# ip link set <interface> mtu <size>
----------------------------

_<interface>_ refers to the network interface used for DRBD
replication. A typical value for _<size>_ would be 9000 (bytes).

[[s-latency-tuning-deadline-scheduler]]
==== Enabling the deadline I/O scheduler

indexterm:[io scheduler]
When used in conjunction with high-performance, write back enabled
hardware RAID controllers, DRBD latency may benefit greatly from using
the simple deadline I/O scheduler, rather than the CFQ scheduler. The
latter is typically enabled by default.

Modifications to the I/O scheduler configuration may be performed via
the `sysfs` virtual file system, mounted at `/sys`. The scheduler
configuration is in `/sys/block/__device__`, where _<device>_ is the
backing device DRBD uses.

Enabling the deadline scheduler works via the following command:
----------------------------
# echo deadline > /sys/block/<device>/queue/scheduler
----------------------------

You may then also set the following values, which may provide
additional latency benefits:

* Disable front merges:
+
----------------------------
# echo 0 > /sys/block/<device>/queue/iosched/front_merges
----------------------------

* Reduce read I/O deadline to 150 milliseconds (the default is 500ms):
+
----------------------------
# echo 150 > /sys/block/<device>/queue/iosched/read_expire
----------------------------

* Reduce write I/O deadline to 1500 milliseconds (the default is
  3000ms):
+
----------------------------
# echo 1500 > /sys/block/<device>/queue/iosched/write_expire
----------------------------

If these values effect a significant latency improvement, you may want
to make them permanent so they are automatically set at system
startup. indexterm:[Debian GNU/Linux]Debian and indexterm:[Ubuntu
Linux]Ubuntu systems provide this functionality via the
`sysfsutils` package and the `/etc/sysfs.conf` configuration file.

You may also make a global I/O scheduler selection by passing the
`elevator` option via your kernel command line. To do so, edit your
boot loader configuration (normally found in `/etc/default/grub` if
you are using the GRUB bootloader) and add `elevator=deadline` to your
list of kernel boot options.
