[[s-using-drbd-proxy]]
== Using DRBD Proxy

[[s-drbd-proxy-deployment-considerations]]
=== DRBD Proxy deployment considerations

The <<s-drbd-proxy,DRBD Proxy>> processes can either be located
directly on the machines where DRBD is set up, or they can be placed
on distinct dedicated servers. A DRBD Proxy instance can serve as a
proxy for multiple DRBD devices distributed across multiple nodes.

DRBD Proxy is completely transparent to DRBD. Typically you will
expect a high number of data packets in flight, therefore the activity
log should be reasonably large. Since this may cause longer re-sync
runs after the crash of a primary node, it is recommended to enable
DRBD's `csums-alg` setting.

For more information about the rationale for the DRBD Proxy, please
see the feature explanation <<s-drbd-proxy>>.

The DRBD Proxy 3 uses several kernel features that are only available since
2.6.26, so running it on older systems (e.g. RHEL 5) is not possible; here
we can still provide DRBD Proxy 1 packages, thoughfootnote:[The v1 uses
a different scheduling model and will therefore not reach the same performance
as v3; so even if your production setup is still RHEL 5, perhaps you can run
one RHEL 6/7 VM in each data center?].


[[s-drbd-proxy-installation]]
=== Installation

To obtain DRBD Proxy, please contact your Linbit sales
representative. Unless instructed otherwise, please always use the
most recent DRBD Proxy release.

To install DRBD Proxy on Debian and Debian-based systems, use the dpkg
tool as follows (replace version with your DRBD Proxy version, and
architecture with your target architecture):

----------------------------
# dpkg -i drbd-proxy_3.2.2_amd64.deb
----------------------------

To install DRBD Proxy on RPM based systems (like SLES or RHEL) use
the rpm tool as follows (replace version with your DRBD Proxy version,
and architecture with your target architecture):

----------------------------
# rpm -i drbd-proxy-3.2.2-1.x86_64.rpm
----------------------------

Also install the DRBD administration program drbdadm since it is
required to configure DRBD Proxy.

This will install the DRBD proxy binaries as well as an init script
which usually goes into `/etc/init.d`. Please always use the init
script to start/stop DRBD proxy since it also configures DRBD Proxy
using the `drbdadm` tool.

[[s-drbd-proxy-license]]
=== License file

When obtaining a license from Linbit, you will be sent a DRBD Proxy
license file which is required to run DRBD Proxy. The file is called
`drbd-proxy.license`, it  must be copied into the `/etc` directory of the
target machines, and be owned by the user/group `drbdpxy`.

----------------------------
# cp drbd-proxy.license /etc/
----------------------------

[[s-drbd-proxy-configuration-linstor]]
=== Configuration using LINSTOR

DRBD Proxy can be configured using LINSTOR as described in the LINSTOR
USER'S GUIDE.

[[s-drbd-proxy-configuration]]
=== Configuration using resource files

DRBD Proxy can also be configured by editing resource files.
It is configured by an additional section called `proxy` and
additional `proxy on` sections within the host sections.

Below is a DRBD configuration example for proxies running directly on
the DRBD nodes:

[source,drbd]
----------------------------
resource r0 {
	protocol A;
	device     /dev/drbd15;
	disk       /dev/VG/r0;
	meta-disk  internal;

	proxy {
		memlimit 512M;
		plugin {
			zlib level 9;
		}
	}

	on alice {
		address 127.0.0.1:7915;
		proxy on alice {
			inside 127.0.0.1:7815;
			outside 192.168.23.1:7715;
		}
	}

	on bob {
		address 127.0.0.1:7915;
		proxy on bob {
			inside 127.0.0.1:7815;
			outside 192.168.23.2:7715;
		}
	}
}
----------------------------

The `inside` IP address is used for communication between DRBD and the
DRBD Proxy, whereas the `outside` IP address is used for communication
between the proxies. The latter channel might have to be allowed in your
firewall setup.

[[s-drbd-proxy-controlling]]
=== Controlling DRBD Proxy

`drbdadm` offers the `proxy-up` and `proxy-down` subcommands to
configure or delete the connection to the local DRBD Proxy process of
the named DRBD resource(s). These commands are used by the `start` and
`stop` actions which `/etc/init.d/drbdproxy` implements.

The DRBD Proxy has a low level configuration tool, called
`drbd-proxy-ctl`. When called without any option it operates in
interactive mode.

To pass a command directly, avoiding interactive mode, use
the `-c` parameter followed by the command.

To display the available commands use:
----------------------------
# drbd-proxy-ctl -c "help"
----------------------------

Note the double quotes around the command being passed.


Here is a list of commands; while the first few ones are typically only used
indirectly (via `drbdadm proxy-up` resp. `drbdadm proxy-down`), the latter ones
give various status information.

.`add connection _<name>_ _lots of arguments_`
Creates a communication path. As this is run via `drbdadm proxy-up` the long list of arguments is omitted here.

.`del connection  _<name>_`
Removes a communication path.

.`set memlimit _<name>_ _<memlimit-in-bytes>_`
Sets the memory limit for a connection; this can only be done when setting it
up afresh, changing it during runtime is not possible. +
This command understands the usual units `k`, `M`, and `G`.

.`show`
Shows currently configured communication paths.

.`show memusage`
Shows memory usage of each connection. +
--
As an example,

--------
# watch -n 1 'drbd-proxy-ctl -c "show memusage"'
--------

monitors memory usage. Please note that the quotes are required as listed above.
--

.`show [h]subconnections`
Shows currently established individual connections
together with some stats. With `h` outputs bytes in human
readable format.

.`show [h]connections`
Shows currently configured connections and their states
With `h` outputs bytes in human readable format. +
--
The column `Status` will show one of these states:

* _Off_: No communication to the remote DRBD Proxy process.
* _Half-up_: The connection to the remote DRBD Proxy could be established; the Proxy => DRBD paths are not up yet.
* _DRBD-conn_: The first few packets are being pushed across the connection; but still e.g. a Split-Brain situation might serve it again.
* _Up_: The DRBD connection is fully established.
--

.`shutdown`
Shuts down the `drbd-proxy` program. Attention: this
unconditionally terminates any DRBD connections using the DRBD proxy.

.`quit`
Exits the client program (closes the control connection), but
leaves the DRBD proxy running.


.`print statistics`
This prints detailed statistics for the currently active connections, in an
easily parseable format. Use this for integration to your monitoring solution! +

NOTE: While the commands above are only accepted from UID 0 (i.e., the `root` user),
this one can be used by any user
(provided that unix permissions allow access on the proxy socket at
`/var/run/drbd-proxy/drbd-proxy-ctl.socket`); see the init script at
`/etc/init.d/drbdproxy` about setting the rights.



[[s-drbd-proxy-plugins]]
=== About DRBD Proxy plugins

Since DRBD Proxy version 3 the proxy allows to enable a few specific
plugins for the WAN connection. +
The currently available plugins are  `zstd`, `lz4`, `zlib` and
`lzma` (all software compression), and `aha` (hardware compression support, see http://www.aha.com/data-compression/).

`zstd` (Zstandard) is a real-time compression algorithm, providing high compression ratios. It offers a very
wide range of compression / speed trade-off, while being backed by a very fast decoder. Compression rates are
dependent on "level" parameter which can be arranged between 1 to 22. Over level 20, Drbd Proxy will require
more memory.

`lz4` is a very fast compression algorithm; the data typically gets compressed
down by 1:2 to 1:4, half- to two-thirds of the bandwidth can be saved.

The `zlib` plugin uses the GZIP algorithm for compression; it uses a bit more CPU than `lz4`, but gives a ratio of 1:3 to 1:5.

The `lzma` plugin uses the `liblzma2` library. It can
use dictionaries of several hundred MiB; these allow for very
efficient delta-compression of repeated data, even for small changes.
`lzma` needs much more CPU and memory, but results in much better
compression than `zlib` -- real-world tests with a VM sitting on top of DRBD
gave ratios of 1:10 to 1:40. The `lzma` plugin has to be enabled in your
license.

`aha` uses hardware compression cards, like the AHA367PCIe (10Gbit/sec) or
AHA372 (20GBit/sec); this is the fastest compression for contemporary hardware. +
You will need a special flag in your license file to enable this plugin.


Please contact LINBIT to find the best settings for your environment - it
depends on the CPU (speed, number of threads), available memory, input and
available output bandwidth, and expected IO spikes. Having a week of `sysstat`
data already available helps in determining the configuration, too.


Please note that the older `compression on` in the
`proxy` section is deprecated, and will be removed in
a future release. +
Currently it is treated as `zlib level 9`.


[[s-drbd-proxy-bwlimit]]
==== Using a WAN Side Bandwidth Limit

The experimental `bwlimit` option of DRBD Proxy is broken. Do not use
it, as it may cause applications on DRBD to block on IO. It will
be removed.

Instead use the Linux kernel's traffic control framework to
limit bandwidth consumed by proxy on the WAN side.

In the following example you would need to replace the interface
name, the source port and the ip address of the peer.

----------------------------
# tc qdisc add dev eth0 root handle 1: htb default 1
# tc class add dev eth0 parent 1: classid 1:1 htb rate 1gbit
# tc class add dev eth0 parent 1:1 classid 1:10 htb rate 500kbit
# tc filter add dev eth0 parent 1: protocol ip prio 16 u32 \
        match ip sport 7000 0xffff \
        match ip dst 192.168.47.11 flowid 1:10
# tc filter add dev eth0 parent 1: protocol ip prio 16 u32 \
        match ip dport 7000 0xffff \
        match ip dst 192.168.47.11 flowid 1:10
----------------------------

You can remove this bandwidth limitation with

----------------------------
# tc qdisc del dev eth0 root handle 1
----------------------------

[[s-drbd-proxy-troubleshoot]]
=== Troubleshooting

DRBD proxy logs via syslog using the `LOG_DAEMON` facility. Usually
you will find DRBD Proxy messages in `/var/log/daemon.log`.

Enabling debug mode in DRBD Proxy can be done with the following command.

--------------------------
# drbd-proxy-ctl -c 'set loglevel debug'
--------------------------

For example, if proxy fails to connect it will log something like
`Rejecting connection because I can't connect on the other side`. In
that case, please check if DRBD is running (not in StandAlone mode) on
both nodes and if both proxies are running. Also double-check your
configuration.
