
            Chapter 18. Using Amanda
Prev  Part IV. Various Information  Next

-------------------------------------------------------------------------------

Chapter 18. Using Amanda


John R. Jackson

Original text
AMANDA Core Team
<jrj@purdue.edu>

Gavin Henry

XML-conversion
Suretec Systems Ltd.
<ghenry@suretecsystems.com>

Stefan G. Weichinger

XML-conversion, Updates
AMANDA Core Team
<sgw@amanda.org>
Table of Contents


  An_Introduction

  Amanda_Features

  Future_Capabilities_of_Amanda

  Amanda_Resources

  Installing_Amanda


        Install_Related_Packages

        Perform_Preliminary_Setup

        Configure_the_Amanda_Build

        Build_and_Install_Amanda

        Configuring_Amanda

        Decide_on_a_Tape_Server

        Decide_Which_Tape_Devices_to_Use

        Decide_Whether_to_Use_Compression

        Decide_Where_the_Holding_Space_Will_Be

        Compute_Your_Dump_Cycle

        Copy_and_Edit_the_Default_Configuration_File

        Configure_the_Holding_Disk

        Configure_Tape_Devices_and_Label_Tapes

        Configure_Backup_Clients

        Test_and_Debug_Setup


  Operating_Amanda


        Run_amdump

        Read_Amanda's_Reports

        Monitor_Tape_and_Holding_Disk_Status

        Adding_Tapes_at_a_Particular_Position_in_the_Cycle

        Miscellanous_Operational_Notes


  Advanced_Amanda_Configuration


        Adjust_the_Backup_Cycle

        Adjust_Parallelism

        Monitor_for_Possible_Improvements

        Excluding_Files


  Restoring_with_Amanda


        Configuring_and_Using_amrecover

        Using_amrestore

        Restoring_Without_Amanda



An Introduction


Note

This chapter was written by John R. Jackson with input from Alexandre Oliva. It
is part of the O'Reilly book "Unix Backup & Recovery" by W. Curtis Preston and
has been provided online at http://www.backupcentral.com/amanda.html since the
first edition of this book.
During the Docbook-conversion of the Amanda-docs we asked for permission to
include this chapter in the Official Amanda documentation and W. Curtis Preston
allowed to us to include the now converted version. There will be some updates
to this chapter in the next few months to reflect various changes and
enhancements.
You can find online versions of this chapter at http://www.amanda.org/docs/
using.html and at http://www.backupcentral.com/amanda.html.
Amanda, the Advanced Maryland Automated Network Disk Archiver, is a public
domain utility developed at the University of Maryland. It is as advanced as a
free backup utility gets, and has quite a large user community. Amanda allows
you to set up a single master backup server to back up multiple hosts to a
single backup drive. (It also works with a number of stackers.) Amanda uses
native dump and/or GNU-tar, and can back up a large number of workstations
running multiple versions of Unix. Recent versions can also use SAMBA to back
up Microsoft Windows (95/98/NT/2000)-based hosts. More information about Amanda
can be found at http://www.amanda.org
Amanda was written primarily by James da Silva at the Department of Computer
Science of the University of Maryland around 1992. The goal was to be able to
back up large numbers of client workstations to a single backup server machine.
Amanda was driven by the introduction of large capacity tape drives, such as
ExaByte 8mm and DAT 4mm. With these drives, and the increased number of
personal workstations, it no longer made sense to back up individual machines
to separate media. Coordinating access and providing tape hardware was
prohibitive in effort and cost. A typical solution to this problem reaches out
to each client from the tape host and dumps areas one by one across the
network. But this usually cannot feed the tape drive fast enough to keep it in
streaming mode, causing a severe performance penalty.

Note

Since Amanda is optimized to take advantage of tape drives, we will use the
word tape throughout this section. However, that doesn't mean that you couldnt
use it with an optical or CD-R drive.
The Amanda approach is to use a "holding disk" on the tape server machine, do
several dumps in parallel into files in the holding disk, and have an
independent process take data out of the holding disk. Because most dumps are
small partials, even a modest amount of holding disk space can provide an
almost optimal flow of dump images onto tape.
Amanda also has a unique approach to scheduling dumps. A "dump cycle" is
defined for each area to control the maximum time between full dumps. Amanda
takes that information, statistics about past dump performance, and estimates
on the size of dumps for this run to decide which backup level to do. This gets
away from the traditional static "it's Friday so do a full dump of /usr on
client A" approach and frees Amanda to balance the dumps so the total run time
is roughly constant from day to day.
Amanda is freely-available software maintained by the Amanda Users Group. Based
on membership of Amanda-related mailing lists, there are probably well over
1500 sites using it. This chapter is based on Amanda version 2.4.2. Updated
versions of this section will be available with the Amanda source code.

Amanda Features

Amanda is designed to handle large numbers of clients and data, yet is
reasonably simple to install and maintain. It scales well, so small
configurations, even a single host, are possible. The code is portable to a
large number of Unix platforms. It calls standard backup software, such as
vendor provided dump or GNU-tar, to perform actual client dumping. There is
also support for backing up Windows-based hosts via SAMBA. There is no
Macintosh support yet.
Amanda provides its own network protocols on top of TCP and UDP. It does not,
for instance, use rsh or rdump/rmt. Each client backup program is instructed to
write to standard output, which Amanda collects and transmits to the tape
server host. This allows Amanda to insert compression and encryption and also
gather a catalogue of the image for recovery. Multiple clients are typically
backed up in parallel to files in one or more holding disk areas. A separate
tape writing process strives to keep the tape device streaming at maximum
throughput. Amanda can run direct to tape without holding disks, but with
reduced performance.
Amanda supports using more than one tape in a single run, but does not yet
split a dump image across tapes. This also means it does not support dump
images larger than a single tape. Amanda currently starts a new tape for each
run and does not provide a mechanism to append a new run to the same tape as a
previous run, which might be an issue for small configurations.
Amanda supports a wide range of tape storage devices. It uses basic operations
through the normal operating system I/O subsystem and a simple definition of
characteristics. New devices are usually trivial to add. Several tape changers,
stackers, and robots are supported to provide truly hands-off operation. The
changer interface is external to Amanda and well-documented, so unsupported
changers can be added without a lot of effort.
Either the client or tape server may do software compression, or hardware
compression may be used. On the client side, software compression reduces
network traffic. On the server side, it reduces client CPU load. Software
compression may be selected on an image-by-image basis. If Kerberos is
available, clients may use it for authentication and dump images may be
encrypted. Without Kerberos, .amandahosts authentication (similar to .rhosts)
is used, or Amanda may be configured to use .rhosts (although rsh/rlogin/rexec
are not themselves used). Amanda works well with security tools like TCP
Wrappers (ftp://info.cert.org/pub/network_tools) and firewalls.
Since standard software is used for generating dump images and software
compression, only normal Unix tools such as mt, dd, and gunzip/uncompress are
needed to recover a dump image from tape if Amanda is not available. When
Amanda software is available, it locates which tapes are needed and finds
images on the tapes.
Amanda is meant to run unattended, such as from a nightly cron job. Client
hosts that are down or hung are noted and bypassed. Tape errors cause Amanda to
fall back to ?degraded? mode where backups are still performed but only to the
holding disks. They may be flushed to tape by hand after the problem is
resolved.
Amanda has configuration options for controlling almost all aspects of the
backup operation and provides several scheduling methods. A typical
configuration does periodic full dumps with partial dumps in between. There is
also support for:

* Periodic archival backup, such as taking full dumps to a vault away from the
  primary site.
* Incremental-only backups where full dumps are done outside of Amanda, such as
  very active areas that must be taken offline, or no full dumps at all for
  areas that can easily be recovered from vendor media.
* Always doing full dumps, such as database areas that change completely
  between each run or critical areas that are easier to deal with during an
  emergency if they are a single-restore operation.

It's easy to support multiple configurations on the same tape server machine,
such as a periodic archival configuration along side a normal daily
configuration. Multiple configurations can run simultaneously on the same tape
server if there are multiple tape drives.
Scheduling of full dumps is typically left up to Amanda. They are scattered
throughout the dump cycle to balance the amount of data backed up each run.
It's important to keep logs of where backup images are for each area (which
Amanda does for you), since they are not on a specific, predictable, tape
(e.g., the Friday tape will not always have a full dump of /usr for client A).
The partial backup level is also left to Amanda. History information about
previous levels is kept and the backup level automatically increases when
sufficient dump size savings will be realized.
Amanda uses a simple tape management system and protects itself from
overwriting tapes that still have valid dump images and from tapes not
allocated to the configuration. Images may be overwritten when a client is down
for an extended period or if not enough tapes are allocated, but only after
Amanda has issued several warnings. Amanda can also be told to not reuse
specific tapes.
A validation program may be used before each run to note potential problems
during normal working hours when they are easier to correct. An activity report
is sent via e-mail after each run. Amanda can also send a report to a printer
and even generate sticky tape labels.
There is no graphical interface. For administration, there is usually only a
single simple text file to edit, so this is not much of an issue. For security
reasons, Amanda does not support user controlled file recovery. There is an
ftp-like restore utility for administrators to make searching online dump
catalogues easier when recovering individual files.

Future Capabilities of Amanda

In addition to the usual enhancements and fixes constantly being added by the
Amanda Core Development Team, three main changes are in various stages of
development.

* A new internal security framework will make it easier for developers to add
  other security methods, such as SSH (ftp://ftp.cs.hut.fi/pub/ssh/) and SSL
  (Secure Socket Layer).
* Another major project is a redesign of how Amanda runs the client dump
  program. This is currently hardcoded for a vendor dump program, GNU-tar or
  SAMBA tar. The new mechanism will allow arbitrary programs such as cpio,
  star, and possibly other backup systems. It will also add optional pre-dump
  and post-dump steps that can be used for locking and unlocking, and snapshots
  of rapidly changing data such as databases or the Windows registry.
* The third major project is a redesign of the output subsystem to support non-
  tape media such as CD-ROM, local files, remote files via tools like rcp and
  ftp, remote tapes, etc. It will also be able to split dump images across
  media, handle multiple simultaneous media of different types such as writing
  to multiple tapes or a tape and a CD-ROM, and handle writing copies of images
  to multiple media such as a tape to keep on site and a CD-ROM or duplicate
  tape for archiving.
* In addition, the output format will be enhanced to include a file-1 and a
  file-n. The idea is to put site-defined emergency recovery tools in file-1
  (the first file on the output) that can be retrieved easily with standard
  non-Amanda programs like tar, then use those tools to retrieve the rest of
  the data. The file-n area is the last file on the output and can contain
  items such as the Amanda database, which would be complete and up to date by
  the time file-n is written.


Amanda Resources

Amanda may be obtained via the web page http://www.amanda.org or with anonymous
ftp at ftp://ftp.amanda.org/pub/amanda.A typical release is a gzip compressed
tar file with a name like amanda-2.4.1.tar.gz, which means it is major version
2.4 and minor version 1. There are occasional patch releases that have a name
like amanda-2.4.1p1.tar.gz (release 2.4.1 plus patch set 1). Beta test pre-
releases have a names like amanda-2.5.0b3.tar.gz (third beta test pre-release
of 2.5.0).
Some operating system distributions provide pre-compiled versions of Amanda,
but because Amanda hardcodes some values into the programs, they may not match
the configuration. Work is being done to move these values to run-time
configuration files, but for now Amanda should be built from source.
The Amanda web page contains useful information about patches not yet part of a
release, how to subscribe to related mailing lists, and pointers to mailing
list archives. Subscribe to at least amanda-announce to get new release
announcements or amanda-users to get announcements plus see problems and
resolutions from other Amanda users. The amanda-users mailing list is a
particularly good resource for help with initial setup as well as problems.
When posting to it, be sure to include the following information:

* Amanda version
* OS version on the server and client(s)
* Exact symptoms seen, such as error messages, relevant sections of e-mail
  reports, debugging and log files
* Anything unusual or recent changes to the environment
* A valid return e-mail address

Finally, the docs directory in the release contains several files with helpful
information, such as a FAQ.

Installing Amanda

After downloading and unpacking the Amanda release, read the README, docs/
INSTALL, and docs/SYSTEM.NOTES files. They contain important and up-to-date
information about how to set up Amanda.

Install Related Packages

Several other packages may be required to complete an Amanda install. Before
continuing, you should locate and install packages your environment will need.
In particular, consider the following:


  GNU-tar 1.12 or later  www.gnu.org
      The GNU version of the standard tar program with enhancements to do
      partial backups and omit selected files. It is one of the client backup
      programs Amanda knows how to use.

  Samba 1.9.18p10 or later  www.samba.org
      SAMBA is an implementation of the System Message Block (SMB) protocol
      used by Windows-based systems for file access. It contains a tool,
      smbclient, that Amanda can use to back them up.

  Perl 5.004 or later  www.perl.org
      Perl is a scripting programming language oriented toward systems
      programming and text manipulation. It is used for a few optional Amanda
      reporting tools and by some tape changers.

  GNU readline 2.2.1 or later  www.gnu.org
      The GNU readline library may be incorporated into interactive programs to
      provide command-line history and editing. It is built into the Amanda
      amrecover restoration tool, if available.

  GNU awk 3.0.3 or later  www.gnu.org
      The GNU version of the awk programming language contains a common version
      across platforms and some additional features. It is used for the
      optional Amanda amplot statistics tool.

  Gnuplot 3.5 or later  ftp://ftp.dartmouth.edu/pub/gnuplot/
      This gnuplot library (which has nothing to do with the GNU tools, see the
      accompanying README) is a graph plotting package. It is used for the
      optional Amanda amplot statistics tool.

Be sure to look in the Amanda patches directory and the patches section on the
web page for updates to these packages. SAMBA versions before 2.0.3, in
particular, must have patches applied to make them work properly with Amanda.
Without the patches, backups appear to work but the resulting images are
corrupt.
When Amanda is configured, locations of additional software used on the
clients, such as GNU-tar and SAMBA, get built into the Amanda programs, so
additional software must be installed in the same place on the Amanda build
machine and all
  Perl 5.0ges

        Perform_Preliminary_Setup

        Configure_the_Amanda_Build

        Build_and_Install_Amanda

        Configuring_Amanda

        Decide_on_a_Tape_Server

        Decide_Which_Tape_Devices_to_Use

        Decide_Whether_to_Use_Compression

        Decide_Where_the_Holding_Space_Will_Be

        Compute_Your_Dump_Cycle

        Copy_and_Edit_the_Default_Configuration_File

        Configure_the_Holding_Disk

        Configure_Tape_Devices_and_Label_Tapes

        Configure_Backup_Clients

        Test_and_Debug_Setup


  Operating_Amanda


        Run_amdump

        Read_Amanda's_Reports

        Monitor_Tape_and_Holding_Disk_Status

        Adding_Tapes_at_a_Particular_Position_in_the_Cycle

        Miscellanous_Operational_Notes


  Advanced_Amanda_Configuration


        Adjust_the_Backup_Cycle

        Adjust_Parallelism

        Monitor_for_Possible_Improvements

        Excluding_Files


  Restoring_with_Amanda


        Configuring_and_Using_amrecover

        Using_amrestore

        Restoring_Without_Amanda



An Introduction


Note

This chapter was written by John R. Jackson with input from Alexandre Oliva. It
is part of the O'Reilly book "Unix Backup & Recovery" by W. Curtis Preston and
has been provided online at http://www.backupcentral.com/amanda.html since the
first edition of this book.
During the Docbook-conversion of the Amanda-docs we asked for permission to
include this chapter in the Official Amanda documentation and W. Curtis Preston
allowed to us to include the now converted version. There will be some updates
to this chapter in the next few months to reflect various changes and
enhancements.
You can find online versions of this chapter at http://www.amanda.org/docs/
using.html and at http://www.backupcentral.com/amanda.html.
Amanda, the Advanced Maryland Automated Network Disk Archiver, is a public
domain utility developed at the University of Maryland. It is as advanced as a
free backup utility gets, and has quite a large user community. Amanda allows
you to set up a single master backup server to back up multiple hosts to a
single backup drive. (It also works with a number of stackers.) Amanda uses
native dump and/or GNU-tar, and can back up a large number of workstations
running multiple versions of Unix. Recent versions can also use SAMBA to back
up Microsoft Windows (95/98/NT/2000)-based hosts. More information about Amanda
can be found at http://www.amanda.org
Amanda was written primarily by James da Silva at the Department of Computer
Science of the University of Maryland around 1992. The goal was to be able to
back up large numbers of client workstations to a single backup server machine.
Amanda was driven by the introduction of large capacity tape drives, such as
ExaByte 8mm and DAT 4mm. With these drives, and the increased number of
personal workstations, it no longer made sense to back up individual machines
to separate media. Coordinating access and providing tape hardware was
prohibitive in effort and cost. A typical solution to this problem reaches out
to each client from the tape host and dumps areas one by one across the
network. But this usually cannot feed the tape drive fast enough to keep it in
streaming mode, causing a severe performance penalty.

Note

Since Amanda is optimized to take advantage of tape drives, we will use the
word tape throughout this section. However, that doesn't mean that you couldnt
use it with an optical or CD-R drive.
The Amanda approach is to use a "holding disk" on the tape server machine, do
several dumps in parallel into files in the holding disk, and have an
independent process take data out of the holding disk. Because most dumps are
small partials, even a modest amount of holding disk space can provide an
almost optimal flow of dump images onto tape.
Amanda also has a unique approach to scheduling dumps. A "dump cycle" is
defined for each area to control the maximum time between full dumps. Amanda
takes that information, statistics about past dump performance, and estimates
on the size of dumps for this run to decide which backup level to do. This gets
away from the traditional static "it's Friday so do a full dump of /usr on
client A" approach and frees Amanda to balance the dumps so the total run time
is roughly constant from day to day.
Amanda is freely-available software maintained by the Amanda Users Group. Based
on membership of Amanda-related mailing lists, there are probably well over
1500 sites using it. This chapter is based on Amanda version 2.4.2. Updated
versions of this section will be available with the Amanda source code.

Amanda Features

Amanda is designed to handle large numbers of clients and data, yet is
reasonably simple to install and maintain. It scales well, so small
configurations, even a single host, are possible. The code is portable to a
large number of Unix platforms. It calls standard backup software, such as
vendor provided dump or GNU-tar, to perform actual client dumping. There is
also support for backing up Windows-based hosts via SAMBA. There is no
Macintosh support yet.
Amanda provides its own network protocols on top of TCP and UDP. It does not,
for instance, use rsh or rdump/rmt. Each client backup program is instructed to
write to standard output, which Amanda collects and transmits to the tape
server host. This allows Amanda to insert compression and encryption and also
gather a catalogue of the image for recovery. Multiple clients are typically
backed up in parallel to files in one or more holding disk areas. A separate
tape writing process strives to keep the tape device streaming at maximum
throughput. Amanda can run direct to tape without holding disks, but with
reduced performance.
Amanda supports using more than one tape in a single run, but does not yet
split a dump image across tapes. This also means it does not support dump
images larger than a single tape. Amanda currently starts a new tape for each
run and does not provide a mechanism to append a new run to the same tape as a
previous run, which might be an issue for small configurations.
Amanda supports a wide range of tape storage devices. It uses basic operations
through the normal operating system I/O subsystem and a simple definition of
characteristics. New devices are usually trivial to add. Several tape changers,
stackers, and robots are supported to provide truly hands-off operation. The
changer interface is external to Amanda and well-documented, so unsupported
changers can be added without a lot of effort.
Either the client or tape server may do software compression, or hardware
compression may be used. On the client side, software compression reduces
network traffic. On the server side, it reduces client CPU load. Software
compression may be selected on an image-by-image basis. If Kerberos is
available, clients may use it for authentication and dump images may be
encrypted. Without Kerberos, .amandahosts authentication (similar to .rhosts)
is used, or Amanda may be configured to use .rhosts (although rsh/rlogin/rexec
are not themselves used). Amanda works well with security tools like TCP
Wrappers (ftp://info.cert.org/pub/network_tools) and firewalls.
Since standard software is used for generating dump images and software
compression, only normal Unix tools such as mt, dd, and gunzip/uncompress are
needed to recover a dump image from tape if Amanda is not available. When
Amanda software is available, it locates which tapes are needed and finds
images on the tapes.
Amanda is meant to run unattended, such as from a nightly cron job. Client
hosts that are down or hung are noted and bypassed. Tape errors cause Amanda to
fall back to ?degraded? mode where backups are still performed but only to the
holding disks. They may be flushed to tape by hand after the problem is
resolved.
Amanda has configuration options for controlling almost all aspects of the
backup operation and provides several scheduling methods. A typical
configuration does periodic full dumps with partial dumps in between. There is
also support for:

* Periodic archival backup, such as taking full dumps to a vault away from the
  primary site.
* Incremental-only backups where full dumps are done outside of Amanda, such as
  very active areas that must be taken offline, or no full dumps at all for
  areas that can easily be recovered from vendor media.
* Always doing full dumps, such as database areas that change completely
  between each run or critical areas that are easier to deal with during an
  emergency if they are a single-restore operation.

It's easy to support multiple configurations on the same tape server machine,
such as a periodic archival configuration along side a normal daily
configuration. Multiple configurations can run simultaneously on the same tape
server if there are multiple tape drives.
Scheduling of full dumps is typically left up to Amanda. They are scattered
throughout the dump cycle to balance the amount of data backed up each run.
It's important to keep logs of where backup images are for each area (which
Amanda does for you), since they are not on a specific, predictable, tape
(e.g., the Friday tape will not always have a full dump of /usr for client A).
The partial backup level is also left to Amanda. History information about
previous levels is kept and the backup level automatically increases when
sufficient dump size savings will be realized.
Amanda uses a simple tape management system and protects itself from
overwriting tapes that still have valid dump images and from tapes not
allocated to the configuration. Images may be overwritten when a client is down
for an extended period or if not enough tapes are allocated, but only after
Amanda has issued several warnings. Amanda can also be told to not reuse
specific tapes.
A validation program may be used before each run to note potential problems
during normal working hours when they are easier to correct. An activity report
is sent via e-mail after each run. Amanda can also send a report to a printer
and even generate sticky tape labels.
There is no graphical interface. For administration, there is usually only a
single simple text file to edit, so this is not much of an issue. For security
reasons, Amanda does not support user controlled file recovery. There is an
ftp-like restore utility for administrators to make searching online dump
catalogues easier when recovering individual files.

Future Capabilities of Amanda

In addition to the usual enhancements and fixes constantly being added by the
Amanda Core Development Team, three main changes are in various stages of
development.

* A new internal security framework will make it easier for developers to add
  other security methods, such as SSH (ftp://ftp.cs.hut.fi/pub/ssh/) and SSL
  (Secure Socket Layer).
* Another major project is a redesign of how Amanda runs the client dump
  program. This is currently hardcoded for a vendor dump program, GNU-tar or
  SAMBA tar. The new mechanism will allow arbitrary programs such as cpio,
  star, and possibly other backup systems. It will also add optional pre-dump
  and post-dump steps that can be used for locking and unlocking, and snapshots
  of rapidly changing data such as databases or the Windows registry.
* The third major project is a redesign of the output subsystem to support non-
  tape media such as CD-ROM, local files, remote files via tools like rcp and
  ftp, remote tapes, etc. It will also be able to split dump images across
  media, handle multiple simultaneous media of different types such as writing
  to multiple tapes or a tape and a CD-ROM, and handle writing copies of images
  to multiple media such as a tape to keep on site and a CD-ROM or duplicate
  tape for archiving.
* In addition, the output format will be enhanced to include a file-1 and a
  file-n. The idea is to put site-defined emergency recovery tools in file-1
  (the first file on the output) that can be retrieved easily with standard
  non-Amanda programs like tar, then use those tools to retrieve the rest of
  the data. The file-n area is the last file on the output and can contain
  items such as the Amanda database, which would be complete and up to date by
  the time file-n is written.


Amanda Resources

Amanda may be obtained via the web page http://www.amanda.org or with anonymous
ftp at ftp://ftp.amanda.org/pub/amanda.A typical release is a gzip compressed
tar file with a name like amanda-2.4.1.tar.gz, which means it is major version
2.4 and minor version 1. There are occasional patch releases that have a name
like amanda-2.4.1p1.tar.gz (release 2.4.1 plus patch set 1). Beta test pre-
releases have a names like amanda-2.5.0b3.tar.gz (third beta test pre-release
of 2.5.0).
Some operating system distributions provide pre-compiled versions of Amanda,
but because Amanda hardcodes some values into the programs, they may not match
the configuration. Work is being done to move these values to run-time
configuration files, but for now Amanda should be built from source.
The Amanda web page contains useful information about patches not yet part of a
release, how to subscribe to related mailing lists, and pointers to mailing
list archives. Subscribe to at least amanda-announce to get new release
announcements or amanda-users to get announcements plus see problems and
resolutions from other Amanda users. The amanda-users mailing list is a
particularly good resource for help with initial setup as well as problems.
When posting to it, be sure to include the following information:

* Amanda version
* OS version on the server and client(s)
* Exact symptoms seen, such as error messages, relevant sections of e-mail
  reports, debugging and log files
* Anything unusual or recent changes to the environment
* A valid return e-mail address

Finally, the docs directory in the release contains several files with helpful
information, such as a FAQ.

Installing Amanda

After downloading and unpacking the Amanda release, read the README, docs/
INSTALL, and docs/SYSTEM.NOTES files. They contain important and up-to-date
information about how to set up Amanda.

Install Related Packages

Several other packages may be required to complete an Amanda install. Before
continuing, you should locate and install packages your environment will need.
In particular, consider the following:


  GNU-tar 1.12 or later  www.gnu.org
      The GNU version of the standard tar program with enhancements to do
      partial backups and omit selected files. It is one of the client backup
      programs Amanda knows how to use.

  Samba 1.9.18p10 or later  www.samba.org
      SAMBA is an implementation of the System Message Block (SMB) protocol
      used by Windows-based systems for file access. It contains a tool,
      smbclient, that Amanda can use to back them up.

  Perl 5.004 or later  www.perl.org
      Perl is a scripting programming language oriented toward systems
      programming and text manipulation. It is used for a few optional Amanda
      reporting tools and by some tape changers.

  GNU readline 2.2.1 or later  www.gnu.org
      The GNU readline library may be incorporated into interactive programs to
      provide command-line history and editing. It is built into the Amanda
      amrecover restoration tool, if available.

  GNU awk 3.0.3 or later  www.gnu.org
      The GNU version of the awk programming language contains a common version
      across platforms and some additional features. It is used for the
      optional Amanda amplot statistics tool.

  Gnuplot 3.5 or later  ftp://ftp.dartmouth.edu/pub/gnuplot/
      This gnuplot library (which has nothing to do with the GNU tools, see the
      accompanying README) is a graph plotting package. It is used for the
      optional Amanda amplot statistics tool.

Be sure to look in the Amanda patches directory and the patches section on the
web page for updates to these packages. SAMBA versions before 2.0.3, in
particular, must have patches applied to make them work properly with Amanda.
Without the patches, backups appear to work but the resulting images are
corrupt.
When Amanda is configured, locations of additional software used on the
clients, such as GNU-tar and SAMBA, get built into the Amanda programs, so
additional software must be installed in the same place on the Amanda build
machine and all
  Perl 5.0ges

        Perform_Preliminary_Setup

        Configure_the_Amanda_Build

        Build_and_Install_Amanda

        Configuring_Amanda

        Decide_on_a_Tape_Server

        Decide_Which_Tape_Devices_to_Use

        Decide_Whether_to_Use_Compression

        Decide_Where_the_Holding_Space_Will_Be

        Compute_Your_Dump_Cycle

        Copy_and_Edit_the_Default_Configuration_File

        Configure_the_Holding_Disk

        Configure_Tape_Devices_and_Label_Tapes

        Configure_Backup_Clients

        Test_and_Debug_Setup


  Operating_Amanda


        Run_amdump

        Read_Amanda's_Reports

        Monitor_Tape_and_Holding_Disk_Status

        Adding_Tapes_at_a_Particular_Position_in_the_Cycle

        Miscellanous_Operational_Notes


  Advanced_Amanda_Configuration


        Adjust_the_Backup_Cycle

        Adjust_Parallelism

        Monitor_for_Possible_Improvements

        Excluding_Files


  Restoring_with_Amanda


        Configuring_and_Using_amrecover

        Using_amrestore

        Restoring_Without_Amanda



An Introduction


Note

This chapter was written by John R. Jackson with input from Alexandre Oliva. It
is part of the O'Reilly book "Unix Backup & Recovery" by W. Curtis Preston and
has been provided online at http://www.backupcentral.com/amanda.html since the
first edition of this book.
During the Docbook-conversion of the Amanda-docs we asked for permission to
include this chapter in the Official Amanda documentation and W. Curtis Preston
allowed to us to include the now converted version. There will be some updates
to this chapter in the next few months to reflect various changes and
enhancements.
You can find online versions of this chapter at http://www.amanda.org/docs/
using.html and at http://www.backupcentral.com/amanda.html.
Amanda, the Advanced Maryland Automated Network Disk Archiver, is a public
domain utility developed at the University of Maryland. It is as advanced as a
free backup utility gets, and has quite a large user community. Amanda allows
you to set up a single master backup server to back up multiple hosts to a
single backup drive. (It also works with a number of stackers.) Amanda uses
native dump and/or GNU-tar, and can back up a large number of workstations
running multiple versions of Unix. Recent versions can also use SAMBA to back
up Microsoft Windows (95/98/NT/2000)-based hosts. More information about Amanda
can be found at http://www.amanda.org
Amanda was written primarily by James da Silva at the Department of Computer
Science of the University of Maryland around 1992. The goal was to be able to
back up large numbers of client workstations to a single backup server machine.
Amanda was driven by the introduction of large capacity tape drives, such as
ExaByte 8mm and DAT 4mm. With these drives, and the increased number of
personal workstations, it no longer made sense to back up individual machines
to separate media. Coordinating access and providing tape hardware was
prohibitive in effort and cost. A typical solution to this problem reaches out
to each client from the tape host and dumps areas one by one across the
network. But this usually cannot feed the tape drive fast enough to keep it in
streaming mode, causing a severe performance penalty.

Note

Since Amanda is optimized to take advantage of tape drives, we will use the
word tape throughout this section. However, that doesn't mean that you couldnt
use it with an optical or CD-R drive.
The Amanda approach is to use a "holding disk" on the tape server machine, do
several dumps in parallel into files in the holding disk, and have an
independent process take data out of the holding disk. Because most dumps are
small partials, even a modest amount of holding disk space can provide an
almost optimal flow of dump images onto tape.
Amanda also has a unique approach to scheduling dumps. A "dump cycle" is
defined for each area to control the maximum time between full dumps. Amanda
takes that information, statistics about past dump performance, and estimates
on the size of dumps for this run to decide which backup level to do. This gets
away from the traditional static "it's Friday so do a full dump of /usr on
client A" approach and frees Amanda to balance the dumps so the total run time
is roughly constant from day to day.
Amanda is freely-available software maintained by the Amanda Users Group. Based
on membership of Amanda-related mailing lists, there are probably well over
1500 sites using it. This chapter is based on Amanda version 2.4.2. Updated
versions of this section will be available with the Amanda source code.

Amanda Features

Amanda is designed to handle large numbers of clients and data, yet is
reasonably simple to install and maintain. It scales well, so small
configurations, even a single host, are possible. The code is portable to a
large number of Unix platforms. It calls standard backup software, such as
vendor provided dump or GNU-tar, to perform actual client dumping. There is
also support for backing up Windows-based hosts via SAMBA. There is no
Macintosh support yet.
Amanda provides its own network protocols on top of TCP and UDP. It does not,
for instance, use rsh or rdump/rmt. Each client backup program is instructed to
write to standard output, which Amanda collects and transmits to the tape
server host. This allows Amanda to insert compression and encryption and also
gather a catalogue of the image for recovery. Multiple clients are typically
backed up in parallel to files in one or more holding disk areas. A separate
tape writing process strives to keep the tape device streaming at maximum
throughput. Amanda can run direct to tape without holding disks, but with
reduced performance.
Amanda supports using more than one tape in a single run, but does not yet
split a dump image across tapes. This also means it does not support dump
images larger than a single tape. Amanda currently starts a new tape for each
run and does not provide a mechanism to append a new run to the same tape as a
previous run, which might be an issue for small configurations.
Amanda supports a wide range of tape storage devices. It uses basic operations
through the normal operating system I/O subsystem and a simple definition of
characteristics. New devices are usually trivial to add. Several tape changers,
stackers, and robots are supported to provide truly hands-off operation. The
changer interface is external to Amanda and well-documented, so unsupported
changers can be added without a lot of effort.
Either the client or tape server may do software compression, or hardware
compression may be used. On the client side, software compression reduces
network traffic. On the server side, it reduces client CPU load. Software
compression may be selected on an image-by-image basis. If Kerberos is
available, clients may use it for authentication and dump images may be
encrypted. Without Kerberos, .amandahosts authentication (similar to .rhosts)
is used, or Amanda may be configured to use .rhosts (although rsh/rlogin/rexec
are not themselves used). Amanda works well with security tools like TCP
Wrappers (ftp://info.cert.org/pub/network_tools) and firewalls.
Since standard software is used for generating dump images and software
compression, only normal Unix tools such as mt, dd, and gunzip/uncompress are
needed to recover a dump image from tape if Amanda is not available. When
Amanda software is available, it locates which tapes are needed and finds
images on the tapes.
Amanda is meant to run unattended, such as from a nightly cron job. Client
hosts that are down or hung are noted and bypassed. Tape errors cause Amanda to
fall back to ?degraded? mode where backups are still performed but only to the
holding disks. They may be flushed to tape by hand after the problem is
resolved.
Amanda has configuration options for controlling almost all aspects of the
backup operation and provides several scheduling methods. A typical
configuration does periodic full dumps with partial dumps in between. There is
also support for:

* Periodic archival backup, such as taking full dumps to a vault away from the
  primary site.
* Incremental-only backups where full dumps are done outside of Amanda, such as
  very active areas that must be taken offline, or no full dumps at all for
  areas that can easily be recovered from vendor media.
* Always doing full dumps, such as database areas that change completely
  between each run or critical areas that are easier to deal with during an
  emergency if they are a single-restore operation.

It's easy to support multiple configurations on the same tape server machine,
such as a periodic archival configuration along side a normal daily
configuration. Multiple configurations can run simultaneously on the same tape
server if there are multiple tape drives.
Scheduling of full dumps is typically left up to Amanda. They are scattered
throughout the dump cycle to balance the amount of data backed up each run.
It's important to keep logs of where backup images are for each area (which
Amanda does for you), since they are not on a specific, predictable, tape
(e.g., the Friday tape will not always have a full dump of /usr for client A).
The partial backup level is also left to Amanda. History information about
previous levels is kept and the backup level automatically increases when
sufficient dump size savings will be realized.
Amanda uses a simple tape management system and protects itself from
overwriting tapes that still have valid dump images and from tapes not
allocated to the configuration. Images may be overwritten when a client is down
for an extended period or if not enough tapes are allocated, but only after
Amanda has issued several warnings. Amanda can also be told to not reuse
specific tapes.
A validation program may be used before each run to note potential problems
during normal working hours when they are easier to correct. An activity report
is sent via e-mail after each run. Amanda can also send a report to a printer
and even generate sticky tape labels.
There is no graphical interface. For administration, there is usually only a
single simple text file to edit, so this is not much of an issue. For security
reasons, Amanda does not support user controlled file recovery. There is an
ftp-like restore utility for administrators to make searching online dump
catalogues easier when recovering individual files.

Future Capabilities of Amanda

In addition to the usual enhancements and fixes constantly being added by the
Amanda Core Development Team, three main changes are in various stages of
development.

* A new internal security framework will make it easier for developers to add
  other security methods, such as SSH (ftp://ftp.cs.hut.fi/pub/ssh/) and SSL
  (Secure Socket Layer).
* Another major project is a redesign of how Amanda runs the client dump
  program. This is currently hardcoded for a vendor dump program, GNU-tar or
  SAMBA tar. The new mechanism will allow arbitrary programs such as cpio,
  star, and possibly other backup systems. It will also add optional pre-dump
  and post-dump steps that can be used for locking and unlocking, and snapshots
  of rapidly changing data such as databases or the Windows registry.
* The third major project is a redesign of the output subsystem to support non-
  tape media such as CD-ROM, local files, remote files via tools like rcp and
  ftp, remote tapes, etc. It will also be able to split dump images across
  media, handle multiple simultaneous media of different types such as writing
  to multiple tapes or a tape and a CD-ROM, and handle writing copies of images
  to multiple media such as a tape to keep on site and a CD-ROM or duplicate
  tape for archiving.
* In addition, the output format will be enhanced to include a file-1 and a
  file-n. The idea is to put site-defined emergency recovery tools in file-1
  (the first file on the output) that can be retrieved easily with standard
  non-Amanda programs like tar, then use those tools to retrieve the rest of
  the data. The file-n area is the last file on the output and can contain
  items such as the Amanda database, which would be complete and up to date by
  the time file-n is written.


Amanda Resources

Amanda may be obtained via the web page http://www.amanda.org or with anonymous
ftp at ftp://ftp.amanda.org/pub/amanda.A typical release is a gzip compressed
tar file with a name like amanda-2.4.1.tar.gz, which means it is major version
2.4 and minor version 1. There are occasional patch releases that have a name
like amanda-2.4.1p1.tar.gz (release 2.4.1 plus patch set 1). Beta test pre-
releases have a names like amanda-2.5.0b3.tar.gz (third beta test pre-release
of 2.5.0).
Some operating system distributions provide pre-compiled versions of Amanda,
but because Amanda hardcodes some values into the programs, they may not match
the configuration. Work is being done to move these values to run-time
configuration files, but for now Amanda should be built from source.
The Amanda web page contains useful information about patches not yet part of a
release, how to subscribe to related mailing lists, and pointers to mailing
list archives. Subscribe to at least amanda-announce to get new release
announcements or amanda-users to get announcements plus see problems and
resolutions from other Amanda users. The amanda-users mailing list is a
particularly good resource for help with initial setup as well as problems.
When posting to it, be sure to include the following information:

* Amanda version
* OS version on the server and client(s)
* Exact symptoms seen, such as error messages, relevant sections of e-mail
  reports, debugging and log files
* Anything unusual or recent changes to the environment
* A valid return e-mail address

Finally, the docs directory in the release contains several files with helpful
information, such as a FAQ.

Installing Amanda

After downloading and unpacking the Amanda release, read the README, docs/
INSTALL, and docs/SYSTEM.NOTES files. They contain important and up-to-date
information about how to set up Amanda.

Install Related Packages

Several other packages may be required to complete an Amanda install. Before
continuing, you should locate and install packages your environment will need.
In particular, consider the following:


  GNU-tar 1.12 or later  www.gnu.org
      The GNU version of the standard tar program with enhancements to do
      partial backups and omit selected files. It is one of the client backup
      programs Amanda knows how to use.

  Samba 1.9.18p10 or later  www.samba.org
      SAMBA is an implementation of the System Message Block (SMB) protocol
      used by Windows-based systems for file access. It contains a tool,
      smbclient, that Amanda can use to back them up.

  Perl 5.004 or later  www.perl.org
      Perl is a scripting programming language oriented toward systems
      programming and text manipulation. It is used for a few optional Amanda
      reporting tools and by some tape changers.

  GNU readline 2.2.1 or later  www.gnu.org
      The GNU readline library may be incorporated into interactive programs to
      provide command-line history and editing. It is built into the Amanda
      amrecover restoration tool, if available.

  GNU awk 3.0.3 or later  www.gnu.org
      The GNU version of the awk programming language contains a common version
      across platforms and some additional features. It is used for the
      optional Amanda amplot statistics tool.

  Gnuplot 3.5 or later  ftp://ftp.dartmouth.edu/pub/gnuplot/
      This gnuplot library (which has nothing to do with the GNU tools, see the
      accompanying README) is a graph plotting package. It is used for the
      optional Amanda amplot statistics tool.

Be sure to look in the Amanda patches directory and the patches section on the
web page for updates to these packages. SAMBA versions before 2.0.3, in
particular, must have patches applied to make them work properly with Amanda.
Without the patches, backups appear to work but the resulting images are
corrupt.
When Amanda is configured, locations of additional software used on the
clients, such as GNU-tar and SAMBA, get built into the Amanda programs, so
additional software must be installed in the same place on the Amanda build
machine and all
  Perl 5.0ges

        Perform_Preliminary_Setup

        Configure_the_Amanda_Build

        Build_and_Install_Amanda

        Configuring_Amanda

        Decide_on_a_Tape_Server

        Decide_Which_Tape_Devices_to_Use

        Decide_Whether_to_Use_Compression

        Decide_Where_the_Holding_Space_Will_Be

        Compute_Your_Dump_Cycle

        Copy_and_Edit_the_Default_Configuration_File

        Configure_the_Holding_Disk

        Configure_Tape_Devices_and_Label_Tapes

        Configure_Backup_Clients

        Test_and_Debug_Setup


  Operating_Amanda


        Run_amdump

        Read_Amanda's_Reports

        Monitor_Tape_and_Holding_Disk_Status

        Adding_Tapes_at_a_Particular_Position_in_the_Cycle

        Miscellanous_Operational_Notes


  Advanced_Amanda_Configuration


        Adjust_the_Backup_Cycle

        Adjust_Parallelism

        Monitor_for_Possible_Improvements

        Excluding_Files


  Restoring_with_Amanda


        Configuring_and_Using_amrecover

        Using_amrestore

        Restoring_Without_Amanda



An Introduction


Note

This chapter was written by John R. Jackson with input from Alexandre Oliva. It
is part of the O'Reilly book "Unix Backup & Recovery" by W. Curtis Preston and
has been provided online at http://www.backupcentral.com/amanda.html since the
first edition of this book.
During the Docbook-conversion of the Amanda-docs we asked for permission to
include this chapter in the Official Amanda documentation and W. Curtis Preston
allowed to us to include the now converted version. There will be some updates
to this chapter in the next few months to reflect various changes and
enhancements.
You can find online versions of this chapter at http://www.amanda.org/docs/
using.html and at http://www.backupcentral.com/amanda.html.
Amanda, the Advanced Maryland Automated Network Disk Archiver, is a public
domain utility developed at the University of Maryland. It is as advanced as a
free backup utility gets, and has quite a large user community. Amanda allows
you to set up a single master backup server to back up multiple hosts to a
single backup drive. (It also works with a number of stackers.) Amanda uses
native dump and/or GNU-tar, and can back up a large number of workstations
running multiple versions of Unix. Recent versions can also use SAMBA to back
up Microsoft Windows (95/98/NT/2000)-based hosts. More information about Amanda
can be found at http://www.amanda.org
Amanda was written primarily by James da Silva at the Department of Computer
Science of the University of Maryland around 1992. The goal was to be able to
back up large numbers of client workstations to a single backup server machine.
Amanda was driven by the introduction of large capacity tape drives, such as
ExaByte 8mm and DAT 4mm. With these drives, and the increased number of
personal workstations, it no longer made sense to back up individual machines
to separate media. Coordinating access and providing tape hardware was
prohibitive in effort and cost. A typical solution to this problem reaches out
to each client from the tape host and dumps areas one by one across the
network. But this usually cannot feed the tape drive fast enough to keep it in
streaming mode, causing a severe performance penalty.

Note

Since Amanda is optimized to take advantage of tape drives, we will use the
word tape throughout this section. However, that doesn't mean that you couldnt
use it with an optical or CD-R drive.
The Amanda approach is to use a "holding disk" on the tape server machine, do
several dumps in parallel into files in the holding disk, and have an
independent process take data out of the holding disk. Because most dumps are
small partials, even a modest amount of holding disk space can provide an
almost optimal flow of dump images onto tape.
Amanda also has a unique approach to scheduling dumps. A "dump cycle" is
defined for each area to control the maximum time between full dumps. Amanda
takes that information, statistics about past dump performance, and estimates
on the size of dumps for this run to decide which backup level to do. This gets
away from the traditional static "it's Friday so do a full dump of /usr on
client A" approach and frees Amanda to balance the dumps so the total run time
is roughly constant from day to day.
Amanda is freely-available software maintained by the Amanda Users Group. Based
on membership of Amanda-related mailing lists, there are probably well over
1500 sites using it. This chapter is based on Amanda version 2.4.2. Updated
versions of this section will be available with the Amanda source code.

Amanda Features

Amanda is designed to handle large numbers of clients and data, yet is
reasonably simple to install and maintain. It scales well, so small
configurations, even a single host, are possible. The code is portable to a
large number of Unix platforms. It calls standard backup software, such as
vendor provided dump or GNU-tar, to perform actual client dumping. There is
also support for backing up Windows-based hosts via SAMBA. There is no
Macintosh support yet.
Amanda provides its own network protocols on top of TCP and UDP. It does not,
for instance, use rsh or rdump/rmt. Each client backup program is instructed to
write to standard output, which Amanda collects and transmits to the tape
server host. This allows Amanda to insert compression and encryption and also
gather a catalogue of the image for recovery. Multiple clients are typically
backed up in parallel to files in one or more holding disk areas. A separate
tape writing process strives to keep the tape device streaming at maximum
throughput. Amanda can run direct to tape without holding disks, but with
reduced performance.
Amanda supports using more than one tape in a single run, but does not yet
split a dump image across tapes. This also means it does not support dump
images larger than a single tape. Amanda currently starts a new tape for each
run and does not provide a mechanism to append a new run to the same tape as a
previous run, which might be an issue for small configurations.
Amanda supports a wide range of tape storage devices. It uses basic operations
through the normal operating system I/O subsystem and a simple definition of
characteristics. New devices are usually trivial to add. Several tape changers,
stackers, and robots are supported to provide truly hands-off operation. The
changer interface is external to Amanda and well-documented, so unsupported
changers can be added without a lot of effort.
Either the client or tape server may do software compression, or hardware
compression may be used. On the client side, software compression reduces
network traffic. On the server side, it reduces client CPU load. Software
compression may be selected on an image-by-image basis. If Kerberos is
available, clients may use it for authentication and dump images may be
encrypted. Without Kerberos, .amandahosts authentication (similar to .rhosts)
is used, or Amanda may be configured to use .rhosts (although rsh/rlogin/rexec
are not themselves used). Amanda works well with security tools like TCP
Wrappers (ftp://info.cert.org/pub/network_tools) and firewalls.
Since standard software is used for generating dump images and software
compression, only normal Unix tools such as mt, dd, and gunzip/uncompress are
needed to recover a dump image from tape if Amanda is not available. When
Amanda software is available, it locates which tapes are needed and finds
images on the tapes.
Amanda is meant to run unattended, such as from a nightly cron job. Client
hosts that are down or hung are noted and bypassed. Tape errors cause Amanda to
fall back to ?degraded? mode where backups are still performed but only to the
holding disks. They may be flushed to tape by hand after the problem is
resolved.
Amanda has configuration options for controlling almost all aspects of the
backup operation and provides several scheduling methods. A typical
configuration does periodic full dumps with partial dumps in between. There is
also support for:

* Periodic archival backup, such as taking full dumps to a                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            