% -*- Mode: latex; -*-
\documentclass[dvipdfm,11pt]{article}
\usepackage[dvipdfm]{hyperref} % Upgraded url package
\parskip=.1in

% Formatting conventions for contributors
% 
% A quoting mechanism is needed to set off things like file names, command
% names, code fragments, and other strings that would confuse the flow of
% text if left undistinguished from preceding and following text.  In this
% document we use the LaTeX macro '\texttt' to indicate such text in the
% source, which normally produces, when used as in '\texttt{special text}',
% the typewriter font.

% It is particularly easy to use this convention if one is using emacs as
% the editor and LaTeX mode within emacs for editing LaTeX documents.  In
% such a case the key sequence ^C^F^T (hold down the control key and type
% 'cft') produces '\texttt{}' with the cursor positioned between the
% braces, ready for the special text to be typed.  The closing brace can
% be skipped over by typing ^e (go to the end of the line) if entering
% text or ^C-} to just move the cursor past the brace.

% LaTeX mode is usually loaded automatically.  At Argonne, one way to 
% get several useful emacs tools working for you automatically is to put
% the following in your .emacs file.

% (require 'tex-site)
% (setq LaTeX-mode-hook '(lambda ()
%                        (auto-fill-mode 1)
%                        (flyspell-mode 1)
%                        (reftex-mode 1)
%                        (setq TeX-command "latex")))

%
% When updating the version number of MPICH2, make sure that you change *all*
% instances of the version.  Search for mpich2- and check each match.  Some
% are within a verbatim statement and will need to be changed for each update.
%

\begin{document}
\markright{MPICH2 Installer's Guide}
\title{{\bf MPICH2 Installer's Guide}\thanks{This work was supported by the
    Mathematical, Information, and Computational Sciences Division
    subprogram of the Office of Advanced Scientific Computing Research,
    SciDAC Program, Office of Science, U.S. Department of Energy, under
    Contract DE-AC02-06CH11357.}\\
  Version %MPICH2_VERSION%\\
  Mathematics and Computer Science Division\\
  Argonne National Laboratory}

\author{
Pavan Balaji\\
Darius Buntinas\\
Ralph Butler\\
Anthony Chan\\
David Goodell\\
William Gropp\\
Jayesh Krishna\\
Rob Latham\\
Ewing Lusk\\
Guillaume Mercier\\
Rob Ross\\
Rajeev Thakur\\[2.0ex]
\textbf{Past Contributors:}\\
David Ashton\\
Brian Toonen
}

\maketitle
\cleardoublepage

\pagenumbering{roman}
\tableofcontents
\clearpage

\pagenumbering{arabic}
\pagestyle{headings}

%% Here is a basic outline for the document.   (Bill's original outline)

%% 0. Quick start with ``best practices''.  Each step has a reference to
%% more detailed information later in the document.
%% 1. Acquiring and unpacking.  Using a ``fast'' directory location and
%%    VPATH
%% 1a. Reporting problems
%% 2. Choosing a device (defer a detailed discussion of each until later)
%% 3. configure, make, and install.  Always use --prefix 
%% show only basic options for configure
%%    3a. Optional include of device-specific information
%%    3b. Optional include of pm-specific information
%%    3c. Optional ``fast'' version
%%    3d. Shared libraries
%% 4. Testing and benchmarking
%% 4a. make testing
%% 4b. Getting, building, and using mpptest and netpipe
%% 5. Special options
%% 6. Troubleshooting
%% Appendix:
%% A. Summary of configure options (particularly the enable and with options)


\section{Introduction}
\label{sec:intro}
This manual describes how to obtain and install MPICH2, the MPI-2
implementation from Argonne National Laboratory.  (Of course, if you are
reading this, chances are good that you have already obtained it and
found this document, among others, in its \texttt{doc} subdirectory.)
This \emph{Guide} will explain how to install MPICH so that you and others can use it to
run MPI applications.  Some particular features are different
if you have system administration privileges (can become ``root'' on a
Unix system), and these are explained here.  It is not necessary to have
such privileges to build and install MPICH2.  In the event of problems,
send mail to \texttt{mpich-discuss@mcs.anl.gov}.  Once MPICH2 is
installed, details on how to run MPI jobs are covered in the \emph{MPICH2
User's Guide}, found in this same \texttt{doc} subdirectory.

MPICH2 has many options.  We will first go through a recommended,
``standard'' installation in a step-by-step fashion, and later describe
alternative possibilities. 


\section{Quick Start}
\label{sec:quick}

In this section we describe a ``default'' set of installation steps.
It uses the default set of configuration options, which builds the
\texttt{nemesis} communication device and the \texttt{Hydra} process
manager, for languages C, C++, Fortran-77, and Fortran-90 (if those
compilers exist), with compilers chosen automatically from the user's
environment, without tracing and debugging options.  It uses the
\texttt{VPATH} feature of \texttt{make}, so that the build process can
take place on a local disk for speed.

\subsection{Prerequisites}
\label{sec:prerequisites}

For the default installation, you will need:
\begin{enumerate}
\item A copy of the distribution, \texttt{mpich2.tar.gz}.
\item A C compiler.
\item A Fortran-77, Fortran-90, and/or C++ compiler if you wish to write
  MPI programs in any of these languages.
\item Any one of a number of Unix operating systems, such as IA32-Linux.
  MPICH2 is most extensively tested on Linux;  there remain some
  difficulties on systems to which we do not currently have access.  Our
  \texttt{configure} script attempts to adapt MPICH2 to new systems. 
\end{enumerate}
Configure will check for these prerequisites and try to work around
deficiencies if possible.  (If you don't have Fortran, you will
still be able to use MPICH2, just not with Fortran applications.)


\subsection{From A Standing Start to Running an MPI Program}
\label{sec:steps}
Here are the steps from obtaining MPICH2 through running your own
parallel program on multiple machines.

\begin{enumerate}
\item 
Unpack the tar file.
\begin{verbatim}
    tar xfz mpich2.tar.gz
\end{verbatim}
If your tar doesn't accept the z option, use
\begin{verbatim}
    gunzip -c mpich2.tar.gz | tar xf -
\end{verbatim}
Let us assume that the directory where you do this is
\texttt{/home/you/libraries}.  It will now contain a subdirectory named
\texttt{mpich2-%MPICH2_VERSION%}.

\item
Choose an installation directory (the default is \texttt{/usr/local/bin)}:
\begin{verbatim}
    mkdir /home/you/mpich2-install
\end{verbatim}
It will be most convenient if this directory is shared by all of the
machines where you intend to run processes.  If not, you will have
to duplicate it on the other machines after installation.  Actually, if
you leave out this step, the next step will create the directory for you.

\item
Choose a build directory.  Building will proceed \emph{much} faster if
your build directory is on a file system local to the machine on which
the configuration and compilation steps are executed.  It is preferable
that this also be separate from the source directory, so that the
source directories remain
clean and can be reused to build other copies on other machines.
\begin{verbatim}
    mkdir /tmp/you/mpich2-%MPICH2_VERSION%
\end{verbatim}

\item
Choose any configure options.  See
Section~\ref{sec:important-configure-options} for a description of the most
important options to consider.  

\item
Configure MPICH2, specifying the installation directory, and running
the \texttt{configure} script in the source directory:
\begin{verbatim}
    cd  /tmp/you/mpich2-%MPICH2_VERSION%
    /home/you/libraries/mpich2-%MPICH2_VERSION%/configure \
            -prefix=/home/you/mpich2-install |& tee c.txt
\end{verbatim}
where the \texttt{$\backslash$} means that this is really one line.  (On
\texttt{sh} and its derivatives, use \verb+2>&1 | tee c.txt+
instead of \verb+|& tee c.txt+).  Other configure options are
described below.  Check the \texttt{c.txt} file to make sure
everything went well.  Problems should be self-explanatory, but if not,
send \texttt{c.txt} to \texttt{mpich-discuss@mcs.anl.gov}.
The file \texttt{config.log} is created by \texttt{configure} and
contains a record of the tests that \texttt{configure} performed.  It
is normal for some tests recorded in \texttt{config.log} to fail.  

\item
Build MPICH2:
\begin{verbatim}
    make |& tee m.txt       (for csh and tcsh)
      OR
    make 2>&1 | tee m.txt   (for bash and sh)
\end{verbatim}
This step should succeed if there were no problems with the preceding
step.  Check file \texttt{m.txt}. If there were problems,
do a \texttt{make clean} and then run make again with \texttt{VERBOSE=1}
\begin{verbatim}
    make VERBOSE=1 |& tee m.txt       (for csh and tcsh)
      OR
    make VERBOSE=1 2>&1 | tee m.txt   (for bash and sh)
\end{verbatim}
and then send \texttt{m.txt} and \texttt{c.txt} to 
\texttt{mpich-discuss@mcs.anl.gov}.

\item
Install the MPICH2 commands:
\begin{verbatim}
    make install |& tee mi.txt
\end{verbatim}
This step collects all required executables and scripts in the \texttt{bin}
subdirectory of the directory specified by the prefix argument to
configure. 

(For users who want an install directory structure compliant to
GNU coding standards (i.e., documentation files go to
\texttt{\$\{datarootdir\}/doc/\$\{PACKAGE\}}, architecture independent
read-only files go to \texttt{\$\{datadir\}/\$\{PACKAGE\}}), replace
\texttt{make install} by

\begin{verbatim}
       make install PACKAGE=mpich2-<versrion>
\end{verbatim}

and corresponding installcheck step should be

\begin{verbatim}
       make installcheck PACKAGE=mpich2-<version>
\end{verbatim}

Setting PACKAGE in \texttt{make install} or \texttt{make installcheck} step
is optional and unnecessary for typical MPI users.)

\item
Add the \texttt{bin} subdirectory of the installation directory to your path:
\begin{verbatim}
    setenv PATH /home/you/mpich2-install/bin:$PATH
\end{verbatim}
for \texttt{csh} and \texttt{tcsh}, or 
\begin{verbatim}
    export PATH=/home/you/mpich2-install/bin:$PATH
\end{verbatim}
for \texttt{bash}, and
\begin{verbatim}
    PATH=/home/you/mpich2-install/bin:$PATH
    export PATH
\end{verbatim}
for \texttt{sh} (the \texttt{bash} syntax may work for more recent 
implementations of \texttt{sh}).  % export var=value is an extension and
                                  % is not in original sh
Check that everything is in order at
this point by doing
\begin{verbatim}
    which mpicc
    which mpiexec
\end{verbatim}
All should refer to the commands in the \texttt{bin} subdirectory of your
install directory.  It is at this point that you will need to
duplicate this directory on your other machines if it is not
in a shared file system. % Just having NFS doesn't mean that directories
                         % are consistencly cross-mounted.

\item Check that you can reach these machines with \texttt{ssh} or
\texttt{rsh} without entering a password.  You can test by doing

\begin{verbatim}
    ssh othermachine date
\end{verbatim}

or

\begin{verbatim}
    rsh othermachine date
\end{verbatim}

If you cannot get this to work without entering a password, you will
need to configure \texttt{ssh} or \texttt{rsh} so that this can be
done.

\item Test the setup you just created:

\begin{verbatim}
    mpiexec -f machinefile -n <number> hostname
\end{verbatim}

The machinefile contains the list of hosts you want to run the
executable on.

\begin{verbatim}
  % cat machinefile
     host1     # Run 1 process on host1
     host2:4   # Run 4 processes on host2
     host3:2   # Run 2 processes on host3
     host4:1   # Run 1 process on host4
\end{verbatim}

\item
Now we will run an MPI job, using the \texttt{mpiexec} command as specified
in the MPI-2 standard.  

As part of the build process for MPICH2, a simple program to compute the value
of $\pi$ by numerical integration is created in the
\texttt{mpich2-%MPICH2_VERSION%/examples} directory.  If the current directory is the top
level MPICH2 build directory, then you can run this program with
\begin{verbatim}
    mpiexec -n 5 -f machinefile ./examples/cpi
\end{verbatim}
The \texttt{cpi} example will tell you which hosts it is running on.

There are many options for \texttt{mpiexec}, by which multiple
executables can be run, hosts can be specified, separate command-line
arguments and environment variables can be passed to different
processes, and working directories and search paths for executables
can be specified.  Do
\begin{verbatim}
    mpiexec --help
\end{verbatim}
for details. A typical example is:
\begin{verbatim}
    mpiexec -f machinefile -n 1 ./master : -n 19 ./slave
\end{verbatim}
to ensure that the process with rank 0 runs on your workstation.

The arguments between `:'s in this syntax are called ``argument sets,''
since they apply to a set of processes.  More argments are described in
the \textit{User's Guide}.

\end{enumerate}

If you have completed all of the above steps, you have successfully
installed MPICH2 and run an MPI example.  

%
% DO NOT REMOVE THE FOLLOWING SECTION.  IF NECESSARY, UPDATE IT.
\subsection{Selecting the Compilers}
\label{sec:compilers}
The MPICH2 configure step will attempt to find the C, C++, and Fortran
compilers for you, but if you either want to override the default or
need to specify a compiler that configure doesn't recognize, you can
specify them on the command line using these variables
\begin{description}
\item[CC]The C compiler.  
\item[CXX]The C++ compiler.  Use \texttt{--disable-cxx} if you do not
  want to build the MPI C++ interface
\item[F77]The Fortran 77 compiler (for the original MPI Fortran
  bindings).  Use \texttt{--disable-f77} if you do not want to build
  either the Fortran 77 or Fortran 90 MPI interfaces
\item[FC]The Fortran 90 (or later) compiler.  Use
  \texttt{--disable-fc} if you do not want to build the Fortran 90 MPI
  interfaces.  Note that in previous versions of MPICH2, the variable
  name was \texttt{F90}.  As Fortran has had 3 major releases since
  Fortran 90 (95, 2003, and 2008), most tools, including those built
  with GNU autotools, have or are changing to use \texttt{FC} instead
  of \texttt{F90}.
\end{description}
For example, to select the Intel compilers instead of the GNU
compilers on a system with both, use
\begin{verbatim}
    ./configure CC=icc CXX=icpc F77=ifort FC=ifort ...
\end{verbatim}
Note the use of the same Fortran compiler, \texttt{ifort}, for both
Fortran 77 and Fortran 90; this is an increasingly common choice.

\subsection{Compiler Optimization Levels}

MPICH2 can be configured with two sets of compiler flags:
\texttt{CFLAGS}, \texttt{CXXFLAGS}, \texttt{FFLAGS}, \texttt{FCFLAGS}
(abbreviated as \texttt{xFLAGS}) and \texttt{MPICH2LIB\_CFLAGS},
\texttt{MPICH2LIB\_CXXFLAGS}, \texttt{MPICH2LIB\_FFLAGS},
\texttt{MPICH2LIB\_FCFLAGS} (abbreviated as \texttt{MPICH2LIB\_xFLAGS})
for compilation; \texttt{LDFLAGS} and \texttt{MPICH2LIB\_LDFLAGS} for linking.
All these flags can be set as part of configure command or 
through environment variables.
(\texttt{CPPFLAGS} stands for C preprocessor flags, which should NOT be set)

Both \texttt{xFLAGS} and \texttt{MPICH2LIB\_xFLAGS} affect the compilation
of the MPICH2 libraries. However, only \texttt{xFLAGS} is appended to 
MPI wrapper scripts, \texttt{mpicc} and friends.

MPICH2 libraries are built with default compiler optimization, \texttt{-O2},
which can be modified by \texttt{--enable-fast} configure option.  For instance,
\texttt{--disable-fast} disables the default optimization option.
\texttt{--enable-fast=O<n>} sets default compiler optimization as
\texttt{-O<n>} (note that this assumes that the compiler accepts this
format). 
For more details of \texttt{--enable-fast}, see the output of
\texttt{configure --help}. 
Any other complicated optimization flags for MPICH2 libraries have
to be set throught \texttt{MPICH2LIB\_xFLAGS}.  \texttt{CFLAGS} and friends
are empty by default.

For example, to build a production MPICH2 environment with \texttt{-O3} for all
language bindings, one can simply do

\begin{verbatim}
  ./configure --enable-fast=all,O3
\end{verbatim}

or

\begin{verbatim}
  ./configure --enable-fast=all MPICH2LIB_CFLAGS=-O3 \
                                MPICH2LIB_FFLAGS=-O3 \
                                MPICH2LIB_CXXFLAGS=-O3 \
                                MPICH2LIB_FCFLAGS=-O3
\end{verbatim}

This will cause the MPICH2 libraries to be built with \texttt{-O3},
and \texttt{-O3} will
not be included in the \texttt{mpicc} and other MPI wrapper script.

There are certain compiler flags that should not be used with MPICH2's
configure, e.g. gcc's \texttt{-Werror} which would confuse configure and cause
certain configure tests to fail to detect the correct system features.
To use \texttt{-Werror} in building MPICH2 libraries, you can pass the compiler
flags during the make step through the Makefile variable,
\texttt{MPICH2\_MAKE\_CFLAGS} as follows:

\begin{verbatim}
  make VERBOSE=1 MPICH2_MAKE_CFLAGS="-Wall -Werror"
\end{verbatim}

(assume CC is set to gcc).  The content of \texttt{MPICH2\_MAKE\_CFLAGS}
is appended to the \texttt{CFLAGS} in almost all Makefiles.



\subsection{Common Non-Default Configuration Options}
\label{sec:non-default}

A brief discussion of some of the \texttt{configure} options is found in
Section~\ref{configure-options}.  Here we comment on some of the most
commonly used options.  
% FIXME: This section needs much more work.

\subsubsection{The Most Important Configure Options}
\label{sec:important-configure-options}
\begin{description}
\item[--prefix]Set the installation directories for MPICH2.  
\item[--enable-debuginfo]Provide access to the message queues for
  debuggers such as Totalview.
\item[--enable-g]Build MPICH2 with various debugging options.  This is
  of interest primarily to MPICH2 developers.  The options
\begin{verbatim}
  --enable-g=dbg,mem,log
\end{verbatim}
 are recommended in that case.
\item[--enable-fast]Configure MPICH2 for fastest performance at the
  expense of error reporting and other program development aids.  This
  is recommended only for getting the best performance out of proven
  production applications, and for benchmarking.
\item[--enable-shared]Build MPICH2 with shared libraries. 
  MPICH2 will try to automatically detect the type of shared library
  support required. See Section~\ref{sec:shared-libraries} for more details.
\item[--with-pm]Select the process manager.  The default is
  \texttt{hydra}; also useful are \texttt{gforker} and \texttt{remshell}.  You can build with
  all three process managers by specifying
\begin{verbatim}
  --with-pm=hydra:gforker:remshell
\end{verbatim}
\item[--without-mpe]Configure MPICH2 without the MPE package of
  program development tools (including the Jumpshot performance viewer)
\item[--with-java]Set the location of Java installation.  This option
is necessary only if the default Java installation in your PATH does
not contain a valid Java installation for Jumpshot, e.g.
\begin{verbatim}
  --with-java=/opt/jdk1.6.0
\end{verbatim}
\end{description}


\subsubsection{Using the Absoft Fortran compilers with MPICH2}
\label{sec:absoft}

For best results, it is important to force the Absoft Fortran compilers to 
make all routine names monocase.  In addition, if lower case is chosen 
(this will match common use by many programs), you must also tell the the 
Absoft compiles to append an underscore to global names in order to access 
routines such as \texttt{getarg} (\texttt{getarg} is not used by MPICH2 but is
used in some 
of the tests and is often used in application programs).  We recommend 
configuring MPICH2 with the following options
\begin{verbatim}

setenv F77 f77
setenv FFLAGS "-f -N15"
setenv FCFLAGS "-YALL_NAMES=LCS -YEXT_SFX=_"

./configure ....

\end{verbatim}




\subsection{Shared Libraries}
\label{sec:shared-libraries}

Shared libraries are currently only supported for gcc (and gcc-like
compilers) on Linux and Mac and for cc on Solaris. To have shared
libraries created when MPICH2 is built, specify the following when
MPICH2 is configured:

\begin{verbatim}
    configure --enable-shared
\end{verbatim}

For users who wish to manually control the linker parameters, this can
be done using:

\begin{verbatim}
    configure --enable-sharedlibs=gcc         (on Linux)
    configure --enable-sharedlibs=osx-gcc     (on Mac OS X)
    configure --enable-sharedlibs=solaris-cc  (on Solaris)
\end{verbatim}




\subsection{What to Tell the Users}
\label{sec:telling}

Now that MPICH2 has been installed, the users have to be informed of how
to use it.  Part of this is covered in the \emph{User's Guide}.  Other
things users need to know are covered here.


\section{Migrating from MPICH1}
\label{sec:migrating}

MPICH2 is an all-new rewrite of MPICH1.  Although the basic steps for
installation have remained the same (\texttt{configure}, \texttt{make},
\texttt{make install}), a number of things have changed.  In this
section we attempt to point out what you may be used to in MPICH1 that
are now different in MPICH2.

\subsection{Configure Options}
\label{sec:configure-options}

The arguments to \texttt{configure} are different in MPICH1 and MPICH2;
the \texttt{Installer's Guide} discusses \texttt{configure}.  In
particular, the newer \texttt{configure} in MPICH2 does not support the
\verb+-cc=<compiler-name>+ (or \texttt{-fc}, \texttt{-c++}, or
\texttt{-f90}) options.  Instead, many of the items that could be
specified in the command line to configure in MPICH1 must now be set by
defining an environment variable.  E.g., while MPICH1 allowed
\begin{verbatim}
    ./configure -cc=pgcc
\end{verbatim}
MPICH2 requires
\begin{verbatim}
    ./configure CC=pgcc
\end{verbatim}
Basically, every
option to the MPICH-1 configure that does not start with
\texttt{--enable} or \texttt{--with} is not available as a configure
option in MPICH2.  Instead, environment variables must be used.  This is
consistent (and required) for use of version 2 GNU \texttt{autoconf}.

\subsection{Other Differences}
Other differences between MPICH1 and MPICH2 include the handling of
process managers and the choice of communication device.

For example, the new process managers have a new format and slightly
different semantics for the \texttt{-machinefile} option.  Assume that
you type this data into a file named \texttt{machfile}:

\begin{verbatim}
    bp400:2
    bp401:2
    bp402:2
    bp403:2
\end{verbatim}

If you then run a parallel job with this machinefile, you would expect
ranks 0 and 1 to run on bp400 because it says to run 2 processes there
before going on to bp401.  Ranks 2 and 3 would run on bp401, and rank
4 on bp402, e.g.:

\begin{verbatim}
    mpiexec -l -machinefile machfile -n 5 hostname
\end{verbatim}

produces:
\begin{verbatim}
    0: bp400
    1: bp400
    2: bp401
    3: bp401
    4: bp402
\end{verbatim}

\section{Choosing the Communication Device}
\label{sec:choose-device}

MPICH2 is designed to be build with many different communication devices,
allowing an implementation to be tuned for different communication fabrics.  A
simple communication device, known as ``ch3'' (for the third version of the
``channel'' interface) is provided with MPICH2 and is the default choice.

The ch3 device itself supports a variety of communication methods.  These are
specified by providing the name of the method after a colon in the
\texttt{--with-device} configure option.  For example,
\texttt{--with-device=ch3:sock} selects the (older) socket-base communication
method.
Methods supported by the MPICH2 group include:

\begin{description}
\item[ch3:nemesis]This method is our new, high performance method. It
  has been made the default communication channel starting the 1.1
  release of MPICH2.  It uses shared-memory to send messages between
  processes on the same node and the network for processes between
  nodes.  Currently sockets and Myrinet-MX are supported networks.  It
  supports \texttt{MPI\_THREAD\_MULTIPLE} and other levels of thread
  safety.
\item[ch3:sock]This method uses sockets for all communications between
  processes. It supports \texttt{MPI\_THREAD\_MULTIPLE} and other
  levels of thread safety.
\end{description}

Most installations should use the default \texttt{ch3:nemesis} method
for best performance. For platforms that are not supported by nemesis,
the \texttt{ch3:sock} method is suggested.

MPICH2 is designed to efficiently support all types of systems.  The
\texttt{ch3:nemesis} device is the primary focus of the MPICH2 group,
but other research groups and computer vendors can and have developed
both their own ch3 ``channels'' as well as complete communication
``devices'' in place of ch3.


%
% Add any configure options that are specific to each method here.  Also
% any environment variables that might apply (e.g., MPICH_INTERFACE_HOSTNAME 
% or MPICH_PORT_RANGE.
%
%\subsection{Special Features of Each Communication Device}
%\subsubsection{ch3:nemesis}

\section{Installing and Managing Process Managers}
\label{sec:process-managers}
MPICH2 has been designed to work with multiple process managers; that
is, although you can start MPICH2 jobs with \texttt{mpiexec}, there are
different mechanisms by which your processes are started.  An interface
(called PMI) isolates the MPICH2 library code from the process manager.
Currently three process managers are distributed with MPICH2
\begin{description}
\item[hydra] This is the default process manager tha natively uses the
existing daemons on the system such as ssh, slurm, pbs.
\item[smpd] This one can be used for both Linux and Windows.  It is the
  only process manager that supports the Windows version of MPICH2.
\item[gforker] This is a simple process manager that creates all
  processes on a single machine.  It is useful both for debugging and
  for running on shared memory multiprocessors.
\end{description}

\subsection{hydra} \label{sec:hydra} \texttt{hydra} is the default
process manager that launches processes using the native daemons
present on the system such as ssh, slurm, pbs, etc. To configure with
the \texttt{hydra} process manager, use

\begin{verbatim}
    configure --with-pm=hydra ...
\end{verbatim}


\subsection{SMPD}
\label{sec:smpd}

\subsubsection{Configuration}
\label{sec:smpd_configure}

You may add the following configure options, 
\texttt{--with-pm=smpd}, 
to build and install the smpd process manager. The process manager, smpd, 
will be installed to the bin sub-directory of the installation directory 
of your choice specified by the \texttt{--prefix} option.

smpd process managers run on each node as stand-alone daemons and need to
be running on all nodes that will participate in MPI jobs.  smpd process 
managers are not connected to each other and rely on a known port to 
communicate with each other.  Note: If you want multiple users to use the 
same nodes they must each configure their smpds to use a unique port per 
user. 

smpd uses a configuration file to store settings.  The default location is 
\verb+~/.smpd+.  This file must not be readable by anyone other than 
the owner and contains at least one required option - the access passphrase.
This is stored in the configuration file as \texttt{phrase=<phrase>}. Access 
to running smpds is authenticated using this passphrase and it must 
not be your user password.

\subsubsection{Usage and administration}
\label{sec:smpd_usage}

Users will start the smpd daemons before launching mpi jobs.  To get an 
smpd running on a node, execute 
\begin{verbatim}
    smpd -s
\end{verbatim}
Executing this for the first time will prompt the user to create a 
\verb+~/.smpd+ configuration file and passphrase if one does not 
already exist.

Then users can use \texttt{mpiexec} to launch MPI jobs.

All options to \texttt{smpd}:

\begin{description}
\item[\texttt{smpd -s}]\mbox{}\\
Start the smpd service/daemon for the current user.  You can add 
\texttt{-p <port>} to specify the port to listen on.  All smpds must use
the same port and if you don't use the default then you will have to
add \texttt{-p <port>} to mpiexec or add the \texttt{port=<port>} to the 
\texttt{.smpd} configuration file.

\item[\texttt{smpd -r}]\mbox{}\\
Start the smpd service/daemon in root/multi-user mode.  This is not yet
implemented.

\item[\texttt{smpd -shutdown [host]}]\mbox{}\\
Shutdown the smpd on the local host or specified host.  Warning: this will
cause the smpd to exit and no mpiexec or smpd commands can be issued to the
host until smpd is started again.

%\item[\texttt{smpd -start}]\mbox{}\\
%Start the Windows smpd service on the local host.
%
%\item[\texttt{smpd -stop}]\mbox{}\\
%Stop the Windows smpd on the local host.
%
%\item[\texttt{smpd -console [host]}]\mbox{}\\
%Connect to a specific smpd to issue console commands.  The currently 
%supported commands are: %get, set, delete, stat, status, shutdown and validate.
%\begin{enumerate}
%\item[\texttt{get <var>}]\mbox{}
%\item[\texttt{set <var>=<value>}]\mbox{}
%\item[\texttt{delete <var>}]\mbox{}
%\item[\texttt{status}]\mbox{}
%\item[\texttt{stat <var>}]\mbox{}
%\item[\texttt{shutdown}]\mbox{}
%\item[\texttt{validate}]\mbox{}
%\end{enumerate}

\end{description}

\subsection{gforker}
\label{sec:forker}
\texttt{gforker} is a simple process manager that runs all processes on
a single node; it's version of \texttt{mpiexec} uses the system
\texttt{fork} and \texttt{exec} calls to create the new processes.
To configure with the \texttt{gforker} process manager, use
\begin{verbatim}
    configure --with-pm=gforker ...
\end{verbatim}

\section{Testing}
\label{sec:testing}
Once MPICH2 has been installed, you can test it by running some of the example
programs in the \texttt{examples} directory.  A more thorough test can be run
with the command \texttt{make testing}.  This will produce a summary on
standard output, along with an XML version of the test results in 
\texttt{mpich2/test/mpi}. In addition, running \texttt{make testing} from the
top-level (\texttt{mpich2}) directory will run tests of the commands, such as
\texttt{mpicc} and \texttt{mpiexec}, that are included with MPICH2.

Other MPI test suites are available from
\url{http://www.mcs.anl.gov/mpi/mpi-test/tsuite.html}.  As part of the MPICH2
development, we run the MPICH1, MPICH2, C++, and Intel test suites every night
and post the results on
\url{http://www.mcs.anl.gov/research/projects/mpich2/nightly/old/}.
Other tests are run on an occasional basis.  

% \subsection{Using the Intel Test Suite}
% \label{sec:intel}

% These instructions are local to our test environment at Argonne.

% How to run a select set of tests from the Intel test suite:

% \begin{small}
% \begin{verbatim}
% 1) checkout the Intel test suite (svn co https://svn.mcs.anl.gov/repos/mpi/test_suites/intel/trunk IntelMPITEST) (outside users
%    can access the most recent version of the test suite from the
%    test suite web page).

% 2) create a testing directory separate from the IntelMPITEST source
% directory

% 3) cd into that testing directory

% 4) make sure the process manager (e.g., mpd) is running

% 5) run "<ITS_SRC_DIR>/configure --with-mpich2=<MPICH2_INSTALL_DIR>", where
% <ITS_SRC_DIR> is the path to the directory Intel test suite source (e.g.,
% /home/toonen/Projects/MPI-Tests/IntelMPITEST) and <MPICH2_INSTALL_DIR> is
% the directory containing your MPICH2 installation

% 6) mkdir Test; cd Test

% 7) find tests in <ITS_SRC_DIR>/{c,fortran} that you are interested in
% running and place the test names in a file.  For example:

% % ( cd /home/toonen/Projects/MPI-Tests/IntelMPITEST/Test ; \
%     find {c,fortran} -name 'node.*' -print | grep 'MPI_Test' 
%     | sed -e 's-/node\..*$--' ) |& tee testlist
% Test/c/nonblocking/functional/MPI_Test
% Test/c/nonblocking/functional/MPI_Testall
% Test/c/nonblocking/functional/MPI_Testany
% Test/c/nonblocking/functional/MPI_Testsome
% Test/c/persist_request/functional/MPI_Test_p
% Test/c/persist_request/functional/MPI_Testall_p
% Test/c/persist_request/functional/MPI_Testany_p
% Test/c/persist_request/functional/MPI_Testsome_p
% Test/c/probe_cancel/functional/MPI_Test_cancelled_false
% Test/fortran/nonblocking/functional/MPI_Test
% Test/fortran/nonblocking/functional/MPI_Testall
% Test/fortran/nonblocking/functional/MPI_Testany
% Test/fortran/nonblocking/functional/MPI_Testsome
% Test/fortran/persist_request/functional/MPI_Test_p
% Test/fortran/persist_request/functional/MPI_Testall_p
% Test/fortran/persist_request/functional/MPI_Testany_p
% Test/fortran/persist_request/functional/MPI_Testsome_p
% Test/fortran/probe_cancel/functional/MPI_Test_cancelled_false
% %

% 8) run the tests using ../bin/mtest:

% % ../bin/mtest -testlist testlist -np 6 |& tee mtest.log
% %

% NOTE: some programs hang if less they are run with less than 6 processes.

% 9) examine the summary.xml file.  look for '<STATUS>fail</STATUS>' to see if
% any failures occurred.  (search for '>fail<' works as well)

% \end{verbatim}
% \end{small}

\section{Benchmarking}
\label{sec:benchmarking}

There are many benchmarking programs for MPI implementations.  Three
that we use are \texttt{mpptest}
(\url{http://www.mcs.anl.gov/mpi/mpptest}), \texttt{netpipe}
(\url{http://www.scl.ameslab.gov/netpipe}), and \texttt{SkaMPI}
(\url{http://liinwww.ira.uka.de/~skampi}).  Each of these has
different strengths and weaknesses and reveals different properties of
the MPI implementation.

In addition, the
MPICH2 test suite contains a few programs to test for performance artifacts in
the directory \texttt{test/mpi/perf}.  An example of a performance artifact is
markedly different performance for the same operation when performed in two
different ways.  For example, using an MPI datatype for a non-contiguous
transfer should not be much slower than packing the data into a contiguous
buffer, sending it as a contiguous buffer, and then unpacking it into the
destination buffer.  An example of this from the MPI-1 standard illustrates
the use of MPI datatypes to transpose a matrix ``on the fly,'' and one test in
\texttt{test/mpi/perf} checks that the MPI implementation performs well in
this case.

\section{MPE}
\label{sec:mpe}

MPICH2 comes with the same MPE (Multi-Processing Environment) tools that are
included with MPICH1.  These include several trace libraries for recording the
execution of MPI programs and the Jumpshot and SLOG tools for performance
visualization.  The MPE tools are built and installed by default and should be
available without requiring any additional steps.  The installation of MPE
is documented in \texttt{mpich2/src/mpe2/INSTALL} and the usage of MPE is
documented in \texttt{mpich2/src/mpe2/README} and MPICH2 user's guide.

\section{Windows Version}
\label{sec:windows}

\subsection{Binary distribution}
\label{sec:winbin}

The Windows binary distribution uses the Microsoft Installer.  Download and 
execute \texttt{mpich2-1.x.xxx.msi} to install the binary distribution.  The default 
installation path is \texttt{C:$\backslash$Program Files$\backslash$MPICH2}. 
You must have administrator privileges to install
\texttt{mpich2-1.x.xxx.msi}.  The installer  
installs a Windows service to launch MPICH applications and only administrators
may install services.  This process manager is called smpd.exe.  Access to 
the process manager is passphrase protected.  The installer asks for this 
passphrase.  Do not use your user password.  The same passphrase must be 
installed on all nodes that will participate in a single MPI job.

Under the installation directory are three sub-directories: \texttt{include},
 \texttt{bin}, and \texttt{lib}.  The \texttt{include} and \texttt{lib} 
directories contain the header files and libraries necessary to compile MPI 
applications.  The \texttt{bin} directory contains the process manager, 
\texttt{smpd.exe}, and the the MPI job launcher, \texttt{mpiexec.exe}.  The
dlls that implement MPICH2 are copied to the Windows system32 directory.

You can install MPICH in unattended mode by executing 
\begin{verbatim}
    msiexec /q /I mpich2-1.x.xxx.msi
\end{verbatim}

The smpd process manager for Windows runs as a service that can launch jobs 
for multiple users.  It does not need to be started like the unix version 
does.  The service is automatically started when it is installed and when 
the machine reboots.  smpd for Windows has additional options:
\begin{description}
\item[\texttt{smpd -start}]\mbox{}\\
Start the Windows \texttt{smpd} service.
\item[\texttt{smpd -stop}]\mbox{}\\
Stop the Windows \texttt{smpd} service.
\item[\texttt{smpd -install}]\mbox{}\\
Install the \texttt{smpd} service.
\item[\texttt{smpd -remove}]\mbox{}\\
Remove the \texttt{smpd} service.
\item[\texttt{smpd -register\_spn}]\mbox{}\\
Register the Service Principal Name with the domain controller.
This command enables passwordless authentication using kerberos.  It must be 
run on each node individualy by a domain administrator.
\end{description}

\subsection{Source distribution}
\label{sec:winsrc}

In order to build MPICH2 from the source distribution under Windows, 
you must have MS Developer Studio .NET 2003 or later, perl and optionally 
Intel Fortran 8 or later.

\begin{itemize}
\item
Download \texttt{mpich2-1.x.y.tar.gz} and unzip it.
\item
Bring up a Visual Studio Command prompt with the compiler environment 
variables set.
\item
Run \texttt{winconfigure.wsf}. If you don't have a Fortran compiler add the 
\texttt{--remove-fortran} option to \texttt{winconfigure} to remove all the Fortran 
projects and dependencies.  Execute \texttt{winconfigure.wsf /?} to see all
available options.
\item 
    open \texttt{mpich2$\backslash$mpich2.sln}
%\item
%    build the \texttt{ch3sockDebug mpich2} solution
%\item
%    build the \texttt{ch3sockDebug mpich2s} project
\item
    build the ch3sockRelease mpich2 solution
\item
    build the ch3sockRelease mpich2s project
%\item
%    build the Debug mpich2 solution
\item
    build the Release mpich2 solution
%\item
%    build the fortDebug mpich2 solution
\item
    build the fortRelease mpich2 solution
%\item
%    build the gfortDebug mpich2 solution
\item
    build the gfortRelease mpich2 solution
%\item
%    build the sfortDebug mpich2 solution
\item
    build the sfortRelease mpich2 solution
\item
    build the channel of your choice.  The options are \texttt{sock} and
    \texttt{nemesis}.
If you plan on launching more processes than you have processors 
you should use the default \texttt{sock} channel.  The \texttt{nemesis} channel 
uses a polling progress engine that can perform poorly when multiple processes 
compete for individual processors.  

\end{itemize}

\subsection{\texttt{cygwin}}
\label{sec:cygwin}

MPICH2 can also be built under \texttt{cygwin} using the source
distribution and the Unix commands described in previous sections.  This
will not build the same libraries as described in this section.  It will 
build a ``Unix'' distribution that runs under \texttt{cygwin}.

It is best to use paths that do not contain embedded spaces when
building MPICH2 under \texttt{cygwin}.  For example, consider using 
\begin{verbatim}
   c:\programFiles\mpich2%MPICH2_VERSION%
\end{verbatim}
instead of 
\begin{verbatim}
   c:\Program Files\mpich2%MPICH2_VERSION%
\end{verbatim}
It may also help to have \texttt{cygwin} installed into a path with no
embedded blanks.

\subsection{\texttt{Vista/Windows 7/Windows Server 2008 Users}}
\label{sec:Vista_install}
To install MPICH version \texttt{1.3} or later please follow the 
instructions provided in \ref{sec:winbin}. Please follow the steps
below to install older versions of MPICH on Windows flavors with User
Account Control.

\begin{enumerate}
\item
    Right-click on the command prompt icon and choose ``Run as administrator''
\item
    You can install MPICH from the administrator command prompt by executing 
\begin{verbatim}
    msiexec /I mpich2-1.x.xxx.msi
\end{verbatim}
\end{enumerate}

\section{All Configure Options}
\label{configure-options}

To get the latest list of all the configure options recognized by the
top-level configure, use:
\begin{verbatim}
    configure --help
\end{verbatim}
Not all of these options may be fully supported yet.  

% Fixme:  Explain some of them, at least the ones that we ask the
% users to try.

\paragraph{Notes on the configure options.}
The \texttt{--with-htmldir} and \texttt{--with-docdir} options specify the 
directories into which the documentation will be installed by \texttt{make
  install}. 

\appendix

\end{document}

%
% Comments on subclassing the document
% We can use \ifdevname ... \fi and \ifpmname ... \fi, as in
% \ifdevchiii .. \fi and \ifpmmpd ... \fi
% (these will still need to be defined)
% There should also be a way to select ``all'' in such a way that the
% document can still flow well, such as
% \ifdevall ... \else \ifdevchiii \else \ifdevmm \fi \fi \fi
