Replication
===========

For backend devices that offer replication features, Cinder provides a common
mechanism for exposing that functionality on a per volume basis while still
trying to allow flexibility for the varying implementation and requirements of
all the different backend devices.

There are 2 sides to Cinder's replication feature, the core mechanism and the
driver specific functionality, and in this document we'll only be covering the
driver side of things aimed at helping vendors implement this functionality in
their drivers in a way consistent with all other drivers.

Although we'll be focusing on the driver implementation there will also be some
mentions on deployment configurations to provide a clear picture to developers
and help them avoid implementing custom solutions to solve things that were
meant to be done via the cloud configuration.

Overview
--------

As a general rule replication is enabled and configured via the cinder.conf
file under the driver's section, and volume replication is requested through
the use of volume types.

*NOTE*: Current replication implementation is v2.1 and it's meant to solve a
very specific use case, the "smoking hole" scenario.  It's critical that you
read the Use Cases section of the spec here:
https://specs.openstack.org/openstack/cinder-specs/specs/mitaka/cheesecake.html

From a user's perspective volumes will be created using specific volume types,
even if it is the default volume type, and they will either be replicated or
not, which will be reflected on the ``replication_status`` field of the volume.
So in order to know if a snapshot is replicated we'll have to check its volume.

After the loss of the primary storage site all operations on the resources will
fail and VMs will no longer have access to the data.  It is then when the Cloud
Administrator will issue the ``failover-host`` command to make the
cinder-volume service perform the failover.

After the failover is completed, the Cinder volume service will start using the
failed-over secondary storage site for all operations and the user will once
again be able to perform actions on all resources that were replicated, while
all other resources will be in error status since they are no longer available.

Storage Device configuration
----------------------------

Most storage devices will require configuration changes to enable the
replication functionality, and this configuration process is vendor and storage
device specific so it is not contemplated by the Cinder core replication
functionality.

It is up to the vendors whether they want to handle this device configuration
in the Cinder driver or as a manual process, but the most common approach is to
avoid including this configuration logic into Cinder and having the Cloud
Administrators do a manual process following a specific guide to enable
replication on the storage device before configuring the cinder volume service.

Service configuration
---------------------

The way to enable and configure replication is common to all drivers and it is
done via the ``replication_device`` configuration option that goes in the
driver's specific section in the ``cinder.conf`` configuration file.

``replication_device`` is a multi dictionary option, that should be specified
for each replication target device the admin wants to configure.

While it is true that all drivers use the same ``replication_device``
configuration option this doesn't mean that they will all have the same data,
as there is only one standardized and **REQUIRED** key in the configuration
entry, all others are vendor specific:

- backend_id:<vendor-identifier-for-rep-target>

Values of ``backend_id`` keys are used to uniquely identify within the driver
each of the secondary sites, although they can be reused on different driver
sections.

These unique identifiers will be used by the failover mechanism as well as in
the driver initialization process, and the only requirement is that is must
never have the value "default".

An example driver configuration for a device with multiple replication targets
is show below::

    .....
    [driver-biz]
    volume_driver=xxxx
    volume_backend_name=biz

    [driver-baz]
    volume_driver=xxxx
    volume_backend_name=baz

    [driver-foo]
    volume_driver=xxxx
    volume_backend_name=foo
    replication_device = backend_id:vendor-id-1,unique_key:val....
    replication_device = backend_id:vendor-id-2,unique_key:val....

In this example the result of calling
``self.configuration.safe_get('replication_device')`` within the driver is the
following list::

    [{backend_id: vendor-id-1, unique_key: val1},
     {backend_id: vendor-id-2, unique_key: val2}]

It is expected that if a driver is configured with multiple replication
targets, that replicated volumes are actually replicated on **all targets**.

Besides specific replication device keys defined in the ``replication_device``,
a driver may also have additional normal configuration options in the driver
section related with the replication to allow Cloud Administrators to configure
things like timeouts.

Capabilities reporting
----------------------

There are 2 new replication stats/capability keys that drivers supporting
replication v2.1 should be reporting: ``replication_enabled`` and
``replication_targets``::

    stats["replication_enabled"] = True|False
    stats["replication_targets"] = [<backend-id_1, <backend-id_2>...]

If a driver is behaving correctly we can expect the ``replication_targets``
field to be populated whenever ``replication_enabled`` is set to ``True``, and
it is expected to either be set to ``[]`` or be missing altogether when
``replication_enabled`` is set to ``False``.

The purpose of the ``replication_enabled`` field is to be used by the scheduler
in volume types for creation and migrations.

As for the ``replication_targets`` field it is only provided for informational
purposes so it can be retrieved through the ``get_capabilities`` using the
admin REST API, but it will not be used for validation at the API layer.  That
way Cloud Administrators will be able to know available secondary sites where
they can failover.

Volume Types / Extra Specs
---------------------------

The way to control the creation of volumes on a cloud with backends that have
replication enabled is, like with many other features, through the use of
volume types.

We won't go into the details of volume type creation, but suffice to say that
you will most likely want to use volume types to discriminate between
replicated and non replicated volumes and be explicit about it so that non
replicated volumes won't end up in a replicated backend.

Since the driver is reporting the ``replication_enabled`` key, we just need to
require it for replication volume types adding ``replication_enabled='<is>
True'`` and also specifying it for all non replicated volume types
``replication_enabled='<is> False'``.

It's up to the driver to parse the volume type info on create and set things up
as requested.  While the scoping key can be anything, it's strongly recommended
that all backends utilize the same key (replication) for consistency and to
make things easier for the Cloud Administrator.

Additional replication parameters can be supplied to the driver using vendor
specific properties through the volume type's extra-specs so they can be used
by the driver at volume creation time, or retype.

It is up to the driver to parse the volume type info on create and retype to
set things up as requested.  A good pattern to get a custom parameter from a
given volume instance is this::

    extra_specs = getattr(volume.volume_type, 'extra_specs', {})
    custom_param = extra_specs.get('custom_param', 'default_value')

It may seem convoluted, but we must be careful when retrieving the
``extra_specs`` from the ``volume_type`` field as it could be ``None``.

Vendors should try to avoid obfuscating their custom properties and expose them
using the ``_init_vendor_properties`` method so they can be checked by the
Cloud Administrator using the ``get_capabilities`` REST API.

*NOTE*: For storage devices doing per backend/pool replication the use of
volume types is also recommended.

Volume creation
---------------

Drivers are expected to honor the replication parameters set in the volume type
during creation, retyping, or migration.

When implementing the replication feature there are some driver methods that
will most likely need modifications -if they are implemented in the driver
(since some are optional)- to make sure that the backend is replicating volumes
that need to be replicated and not replicating those that don't need to be:

- ``create_volume``
- ``create_volume_from_snapshot``
- ``create_cloned_volume``
- ``retype``
- ``clone_image``
- ``migrate_volume``

In these methods the driver will have to check the volume type to see if the
volumes need to be replicated, we could use the same pattern described in the
`Volume Types / Extra Specs`_ section::

    def _is_replicated(self, volume):
        specs = getattr(volume.volume_type, 'extra_specs', {})
        return specs.get('replication_enabled') == '<is> True'

But it is **not** the recommended mechanism, and the ``is_replicated`` method
available in volumes and volume types versioned objects instances should be
used instead.

Drivers are expected to keep the ``replication_status`` field up to date and in
sync with reality, usually as specified in the volume type.  To do so in above
mentioned methods' implementation they should use the update model mechanism
provided for each one of those methods.  One must be careful since the update
mechanism may be different from one method to another.

What this means is that most of these methods should be returning a
``replication_status`` key with the value set to ``enabled`` in the model
update dictionary if the volume type is enabling replication.  There is no need
to return the key with the value of ``disabled`` if it is not enabled since
that is the default value.

In the case of the ``create_volume``, and ``retype`` method there is no need to
return the ``replication_status`` in the model update since it has already been
set by the scheduler on creation using the extra spec from the volume type. And
on ``migrate_volume`` there is no need either since there is no change to the
``replication_status``.

*NOTE*: For storage devices doing per backend/pool replication it is not
necessary to check the volume type for the ``replication_enabled`` key since
all created volumes will be replicated, but they are expected to return the
``replication_status`` in all those methods, including the ``create_volume``
method since the driver may receive a volume creation request without the
replication enabled extra spec and therefore the driver will not have set the
right ``replication_status`` and the driver needs to correct this.

Besides the ``replication_status`` field that drivers need to update there are
other fields in the database related to the replication mechanism that the
drivers can use:

- ``replication_extended_status``
- ``replication_driver_data``

These fields are string type fields with a maximum size of 255 characters and
they are available for drivers to use internally as they see fit for their
normal replication operation.  So they can be assigned in the model update and
later on used by the driver, for example during the failover.

To avoid using magic strings drivers must use values defined by the
``ReplicationStatus`` class in ``cinder/objects/fields.py`` file and
these are:

- ``ERROR``: When setting the replication failed on creation, retype, or
  migrate.  This should be accompanied by the volume status ``error``.
- ``ENABLED``: When the volume is being replicated                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          