.. raw:: html

  <a href="https://dials.github.io/dials-2.2/documentation/tutorials/multi_crystal_symmetry_and_scaling.html" class="new-documentation">
  This tutorial requires a DIALS 3 installation.<br/>
  Please click here to go to the tutorial for DIALS 2.2.
  </a>

Multi-crystal symmetry analysis and scaling with DIALS
======================================================

Introduction
------------

Recent additions to DIALS and xia2 have enabled multi-crystal analysis to be
performed following integration. These tools are particularly relevant
for analysis of many partial-datasets, which may be the only practical way of
performing data collections for certain crystals. After integration, the
space group symmetry can be investigated by testing for the presence of symmetry
operations relating the integrated intensities of groups of reflections - the
program to perform this is analysis is :samp:`dials.symmetry` (with algorithms
similar to those of the program Pointless_).
Another thing to consider is that for certain space groups (polar space groups),
there is an inherent ambiguity in the way that the diffraction pattern can be
indexed. In order to combine multiple datasets from these space groups, one must
reindex all data to a consistent setting, which can be done with the program
:samp:`dials.cosym` (see `Gildea and Winter`_ for details).
Finally, the data must be scaled, to correct for experimental effects such as
differences in crystal size/illuminated volume and radiation damage - this can
be done with the program :samp:`dials.scale` (with algorithms similar to those
of the program Aimless_). After the data has been scaled, choices
can then be made about applying a resolution limit to exclude certain regions
of the data which may be negatively affected by radiation damage.

In this tutorial, we shall investigate a multi-crystal dataset collected on
the VMXi beamline, Diamond's automated facility for data collection from
crystallisation experiments *in-situ*. The dataset consists of four repeats of
a 60-degree rotation measurement on a crystal of Proteinase K, taken at different
locations on the crystal. We shall start with the integrated reflections and
experiments files generated by running the automated processing software
:samp:`xia2` with :samp:`pipeline=dials`.
Have a look at the :doc:`processing_in_detail_betalactamase` tutorial if you
want to know more about the different processing steps up to this point.

..  Note::
    To obtain the data for this tutorial you can run
    :samp:`dials.data get vmxi_proteinase_k_sweeps`. If you are at Diamond
    Light Source on BAG training then the data are already available.
    After typing :samp:`module load bagtraining` you'll be moved to a working
    folder, with the data already located in the :samp:`tutorial-data/ccp4/integrated_files`
    subdirectory. The processing in this tutorial will produce quite a few files,
    so it's recommended to make an move to new directory::

      mkdir multi_crystal
      cd multi_crystal


xia2.multiplex
--------------
The easiest way to run these tools for a multi-dataset analysis is through the
program :samp:`xia2.multiplex`.
This runs several DIALS programs, including the programs described above, while
producing useful plots and output files.

To run :samp:`xia2.multiplex`, we must provide the path to the input integrated files from
:samp:`dials.integrate`:

.. dials_tutorial_include:: multi_crystal/xia2.multiplex.cmd

.. container:: toggle

    .. container:: header

        **Show/Hide Log**

    ..  dials_tutorial_include:: multi_crystal/xia2.multiplex.log

This runs :samp:`dials.cosym` to analyse the Laue symmetry and reindex all datasets
consistently, scales the data with :samp:`dials.scale`,
calculates a resolution limit with :samp:`dials.estimate_resolution` and reruns
:samp:`dials.scale` with the determined resolution cutoff. The
final dataset is exported to an unmerged mtz and a
`HTML report <https://dials.github.io/tutorial_data/master/multi_crystal/xia2.multiplex.html>`_
is generated. The easiest way to see the results is to open the
`HTML report <https://dials.github.io/tutorial_data/master/multi_crystal/xia2.multiplex.html>`_
in your browser of choice e.g.::

  firefox xia2.multiplex.html

Provided is a summary of the merging statistics as well as several plots, please
explore these for a few minutes now!
This dataset results in good merging statistics, however if you navigate to the
"Analysis by batch" tab in "All data", you will see that the fourth dataset has
poorer statistics compared to the others. Let's repeat the processing manually
to explore the different steps and address this issue.

Manual reprocessing
-------------------
The first step is Laue/Patterson group analysis using
:doc:`dials.cosym <../programs/dials_cosym>`:

.. dials_tutorial_include:: multi_crystal/dials.cosym.cmd

.. dials_tutorial_include:: multi_crystal/dials.cosym.log
   :start-at: Scoring all possible sub-groups
   :end-before: Writing html report

.. container:: toggle

    .. container:: header

        **Show/Hide Log**

    .. dials_tutorial_include:: multi_crystal/dials.cosym.log


As you can see, the :math:`P\,4/m\,m\,m` Patterson group is found with the highest confidence.
For the corresponding space group, the mirror symmetries are removed to give :math:`P\,4\,2\,2`,
as the chiral nature of macromolecules means we have a restricted choice of space
groups. In this example, all datasets were indexed consistently, but this is not
the case in general.

Next, the data can be scaled:

.. dials_tutorial_include:: multi_crystal/dials.scale.cmd

From the merging statistics it is clear that the data quality is good out to the
furthest resolution (:math:`CC_{1/2} > 0.3`), which can be confirmed by a resolution analysis:

.. dials_tutorial_include:: multi_crystal/dials.estimate_resolution.cmd

.. dials_tutorial_include:: multi_crystal/dials.estimate_resolution.log
   :start-at: Resolution cc_half
   :end-at: Resolution cc_half

.. container:: toggle

    .. container:: header

        **Show/Hide Log**

    .. dials_tutorial_include:: multi_crystal/dials.estimate_resolution.log


If the resolution limit was lower than the extent of the data, scaling would
be rerun with a new resolution limit, for example:

.. dials_tutorial_include:: multi_crystal/dials.scale_cut.cmd

.. container:: toggle

    .. container:: header

        **Show/Hide Log**

    .. dials_tutorial_include:: multi_crystal/dials.scale_cut.log

For exploring the scaling results, a wide variety of scaling and merging plots
can be found in the :samp:`dials.scale.html` report generated by :samp:`dials.scale`.

Almost there
------------
As mentioned previously, the fourth dataset is giving significantly higher
R-merge values and much lower I/sigma.
Therefore the question one must ask is if it is better to exclude this dataset.
We can get some useful information about the agreement between datasets by
running the program :samp:`dials.compute_delta_cchalf`. This program implements
a version of the algorithms described in Assmann_ *et al.* :

.. dials_tutorial_include:: multi_crystal/dials.compute_delta_cchalf.cmd

.. dials_tutorial_include:: multi_crystal/dials.compute_delta_cchalf.log
   :start-at: # Datasets:
   :end-before: Writing table

.. container:: toggle

    .. container:: header

        **Show/Hide Log**

    .. dials_tutorial_include:: multi_crystal/dials.compute_delta_cchalf.log

It looks like we could get a significantly better :math:`CC_{1/2}` by excluding the final
dataset - it has a negative :math:`\Delta CC_{1/2}`. But how bad is too bad that it warrants
exclusion? Unfortunately this is a difficult question to answer and it may be the
case that one would need to refine several structures with different data excluded
to properly address this question.
If we had many datasets and only a small fraction had a very large negative :math:`\Delta CC_{1/2}`
then one could argue that these measurements are not drawn from the same population
as the rest of the data and should be excluded.

To see the effect of removing the last dataset (dataset '3'), we can rerun
:samp:`dials.scale` (note that this will overwrite the previous scaled files):

.. dials_tutorial_include:: multi_crystal/dials.scale_exclude.cmd

.. container:: toggle

    .. container:: header

        **Show/Hide Log**

    .. dials_tutorial_include:: multi_crystal/dials.scale_exclude.log

The overall merging statistics look significantly improved and therefore
one would probably proceed with the first three datasets::

  Resolution:         68.40 - 1.78  > 68.40 - 1.79
  Observations:       222563        > 166095
  Unique reflections: 16534         > 16285
  Redundancy:         13.5          > 10.2
  Completeness:       68.18%        > 67.56%
  Mean intensity:     45.3          > 46.0
  Mean I/sigma(I):    25.0          > 26.1
  R-merge:            0.132         > 0.059
  R-meas:             0.136         > 0.062
  R-pim:              0.033         > 0.017


We could have also excluded a subset of images, for example using the option
:samp:`exclude_images=3:301:600` to exclude the last 300 images of dataset 3.
This option could be used to exclude the end of a dataset that was showing
sigificant radiation damage, or if the crystal had moved out of the beam part-way
through the measurement.

It is also worth checking the assigned space group using :samp:`dials.symmetry`.
In ``dials.cosym``, only the Laue/Patterson group was tested to determine a space
group of :math:`P\,4\,2\,2`. However, a number of other MX space groups are possible for the
Laue group (due to the possibility of screw-axes), such as :math:`P\,4\,2_1\,2`,
:math:`P\,4_1\,2\,2` etc. The screw-axes tests are performed by :samp:`dials.symmetry`, and we can disable the
Laue group testing as we are already confident about this:

.. dials_tutorial_include:: multi_crystal/dials.symmetry.cmd

.. dials_tutorial_include:: multi_crystal/dials.symmetry.log
   :start-after: Laue group
   :end-before: Saving reindexed experiments

.. container:: toggle

    .. container:: header

        **Show/Hide Log**

    .. dials_tutorial_include:: multi_crystal/dials.symmetry.log

By analysing the sets of reflections we expect to be present and absent, the
existence of the :math:`4_1` and :math:`2_1`  screw axes are confirmed, hence the space group is
assigned as :math:`P\,4_1\,2_1\,2`.
Note that we can do this analysis before or after scaling, as we only need to know
the Laue group for scaling, however it is preferable to do this after scaling as
outliers may have been removed by scaling.

Finally, we must merge the data and produce an MTZ file for downstream structure
solution:

.. dials_tutorial_include:: multi_crystal/dials.merge.cmd

.. container:: toggle

    .. container:: header

        **Show/Hide Log**

    .. dials_tutorial_include:: multi_crystal/dials.merge.log

This merges the data and performs a truncation procedure, to give a merged MTZ
file containing intensities and strictly-positive structure factors (Fs).


.. _Pointless: http://www.ccp4.ac.uk/html/pointless.html
.. _`Gildea and Winter`: https://doi.org/10.1107/S2059798318002978
.. _Aimless: http://www.ccp4.ac.uk/html/aimless.html
.. _Assmann: https://doi.org/10.1107/S1600576716005471
