
                     How to configure to use "make check"

What is "make check"

   "make check" is a target in the top level makefile. It takes care of
   running a number of unit and system tests to confirm that FreeSWAN has
   been compiled correctly, and that no new bugs have been introduced.

   As FreeSWAN contains both kernel and userspace components, doing
   testing of FreeSWAN requires that the kernel be simulated. This is
   typically difficult to do as a kernel requires that it be run on bare
   hardware. A technology has emerged that makes this simpler. This is
   [1]User Mode Linux.

   User-Mode Linux is a way to build a Linux kernel such that it can run
   as a process under another Linux (or in the future other) kernel.
   Presently, this can only be done for 2.4 guest kernels. The host
   kernel can be 2.2 or 2.4.

   "make check" expects to be able to build User-Mode Linux kernels with
   FreeSWAN included. To do this it needs to have some files downloaded
   and extracted prior to running "make check". This is described in the
   [2]UML testing document.

   After having run the example in the UML testing document and
   successfully brought up the four machine combination, you are ready to
   use "make check"

Running "make check"

   "make check" works by walking the FreeSWAN source tree invoking the
   "check" target at each node. At present there are tests defined only
   for the klips directory. These tests will use the UML infrastructure
   to test out pieces of the klips code.

   The results of the tests can be recorded. If the environment variable
   $REGRESSRESULTS is non-null, then the results of each test will be
   recorded. This can be used as part of a nightly regression testing
   system, see [3]Nightly testing for more details.

   "make check" otherwise prints a minimal amount of output for each
   test, and indicates pass/fail status of each test as they are run.
   Failed tests do not cause failure of the target in the form of exit
   codes.

                       How to write a "make check" test

Structure of a test

   Each test consists of a set of directories under testing/. There are
   directories for klips, pluto, packaging and libraries. Each directory
   has a list of tests to run is stored in a file called TESTLIST in that
   directory. e.g. testing/klips/TESTLIST.

The TESTLIST

   This isn't actually a shell script. It just looks like one. Some tools
   other than /bin/sh process it. Lines that start with # are comments.
# test-kind     directory-containing-test       expectation     [PR#]

   The first word provides the test type, detailed below.

   The second word is the name of the test to run. This the directory in
   which the test case is to be found..

   The third word may be one of:

   blank/good
          the test is believed to function, report failure

   bad
          the test is known to fail, report unexpected success

   suspended
          the test should not be run

   The fourth word may be a number, which is a PR# if the test is
   failing.

Test kinds

   The test types are:

   skiptest
          means run no test.

   ctltest
          means run a single system without input/output.

   klipstest
          means run a single system with input/output networks

   [4]umlplutotest
          means run a pair of systems

   [5]umlXhost
          run an arbitrary number of systems

   suntest (TBD)
          means run a quad of east/west/sunrise/sunset

   roadtest (TBD)
          means run a trio of east-sunrise + warrior

   extrudedtest (TBD)
          means run a quad of east-sunrise + warriorsouth + park

   mkinsttest
          a test of the "make install" machinery.

   kernel_test_patch
          a test of the "make kernelpatch" machinery.

   Tests marked (TBD) have yet to be fully defined.

   Each test directory has a file in it called testparams.sh. This file
   sets a number of environment variables to define the parameters of the
   test.

Common parameters

   TESTNAME
          the name of the test (repeated for checking purposes)

   TEST_TYPE
          the type of the test (repeat of type type above)

   TESTHOST
          the name of the UML machine to run for the test, typically
          "east" or "west"

   TEST_PURPOSE
          The purpose of the test is one of:

        goal
                The goal purpose is where a test is defined for code that
                is not yet finished. The test indicates when the goals
                have in fact been reached.

        regress
                This is a test to determine that a previously existing
                bug has been repaired. This test will initially be
                created to reproduce the bug in isolation, and then the
                bug will be fixed.

        exploit
                This is a set of packets/programs that causes a
                vulnerability to be exposed. It is a specific variation
                of the regress option.

   TEST_GOAL_ITEM
          in the case of a goal test, this is a reference to the
          requirements document

   TEST_PROB_REPORT
          in the case of regression test, this the problem report number
          from GNATS

   TEST_EXPLOIT_URL
          in the case of an exploit, this is a URL referencing the paper
          explaining the origin of the test and the origin of exploit
          software

   REF_CONSOLE_OUTPUT
          a file in the test directory that contains the sanitized
          console output against which to compare the output of the
          actual test.

   REF_CONSOLE_FIXUPS
          a list of scripts (found in klips/test/fixups) to apply to
          sanitize the console output of the machine under test. These
          are typically perl, awk or sed scripts that remove things in
          the kernel output that change each time the test is run and/or
          compiled.

   INIT_SCRIPT
          a file of commands that is fed into the virtual machine's
          console in single user mode prior to starting the tests. This
          file will usually set up any eroute's and SADB entries that are
          required for the test.

          Lines beginning with # are skipped. Blank lines are skipped.
          Otherwise, a shell prompted is waited for each time (consisting
          of \n#) and then the command is sent. Note that the prompt is
          waited for before the command and not after, so completion of
          the last command in the script is not required. This is often
          used to invoke a program to monitor the system, e.g. ipsec
          pf_key.

   RUN_SCRIPT
          a file of commands that is fed into the virtual machine's
          console in single user mode, before the packets are sent. On
          single machine tests, this script doesn't provide any more
          power than INIT_SCRIPT, but is implemented for consistency's
          sake.

   FINAL_SCRIPT
          a file of commands that is fed into the virtual machine's
          console in single user mode after the final packet is sent.
          Similar to INIT_SCRIPT, above. If not specified, then the
          single command "halt" is sent. If specified, then the script
          should end with a halt command to nicely shutdown the UML.

   CONSOLEDIFFDEBUG
          If set to "true" then the series of console fixups (see
          REF_CONSOLE_FIXUPS) will be output after it is constructed. (It
          should be set to "false", or unset otherwise)

   NETJIGDEBUG
          If set to "true" then the series of console fixups (see
          REF_CONSOLE_FIXUPS) will be output after it is constructed. (It
          should be set to "false", or unset otherwise)

   NETJIGTESTDEBUG
          If set to "netjig", then the results of talking to the
          uml_netjig will be printed to stderr during the test. In
          addition, the jig will be invoked with --debug, which causes it
          to log its process ID, and wait 60 seconds before continuing.
          This can be used if you are trying to debug the uml_netjig
          program itself.

   HOSTTESTDEBUG
          If set to "hosttest", then the results of taling to the
          consoles of the UMLs will be printed to stderr during the test.

   NETJIGWAITUSER
          If set to "waituser", then the scripts will wait forever for
          user input before they shut the tests down. Use this is if you
          are debugging through the kernel.

   PACKETRATE
          A number, in miliseconds (default is 500ms) at which packets
          will be replayed by the netjig.

mkinsttest paramaters

   The basic concept of the mkinsttest test type is that it performs a
   "make install" to a temporary $DESTDIR. The resulting tree can then be
   examined to determine if it was done properly. The files can be
   uninstalled to determine if the file list was correct, or the contents
   of files can be examined more precisely.

   INSTALL_FLAGS
          If set, then an install will be done. This provides the set of
          flags to provide for the install. The target to be used
          (usually "install") must be among the flags.

   POSTINSTALL_SCRIPT
          If set, a script to run after initial "make install". Two
          arguments are provided: an absolute path to the root of the
          FreeSWAN src tree, and an absolute path to the temporary
          installation area.

   INSTALL2_FLAGS
          If set, a second install will be done using these flags.
          Similarly to INSTALL_FLAGS, the target must be among the flags.

   UNINSTALL_FLAGS
          If set, an uninstall will be done using these flags. Similarly
          to INSTALL_FLAGS, the target (usually "uninstall") must be
          among the flags.

   REF_FIND_f_l_OUTPUT
          If set, a find $ROOT ( -type f -or -type -l ) will be done to
          get a list of a real files and symlinks. The resulting file
          will be compared to the file listed by this option.

   REF_FILE_CONTENTS
          If set, it should point to a file containing records for the
          form:

  reffile   samplefile

          one record per line. A diff between the provided reference
          file, and the sample file (located in the temporary
          installation root) will be done for each record.

rpm_build_install_test paramaters

   The rpm_build_install_test type is to verify that the proper packing
   list is produced by "make rpm", and that the mechanisms for building
   the kernel modules produce consistent results.

   RPM_KERNEL_SOURCE
          Point to an extracted copy of the RedHat kernel source code.
          Variables from the environment may be used.

   REF_RPM_CONTENTS
          This is a file containing one record per line. Each record
          consists of a RPM name (may contain wildcards) and a filename
          to compare the contents to. The RPM will be located and a file
          list will be produced with rpm2cpio.

libtest paramaters

   The libtest test is for testing library routines. The library file is
   expected to provided an #ifdef by the name of library
   . The libtest type invokes the C compiler to compile this file, links
   it against libfreeswan.a (to resolve any other dependancies) and runs
   the test with the -r argument to invoke a regression test.

   The library test case is expected to do a self-test, exiting with
   status code 0 if everything is okay, and with non-zero otherwise. A
   core dump (exit code greater than 128) is noted specifically.

   Unlike other tests, there are no subdirectories required, or other
   parameters to set.

umlplutotest paramaters

   The umlplutotest function starts a pair of user mode line processes.
   This is a 2-host version of umlXhost. The "EAST" and "WEST" slots are
   defined.

umlXhost parameters

   The umlXtest function starts an arbitrary number of user mode line
   processes.

   The script invoked (testing/utils/Xhost-test.tcl) is a TCL [9]expect
   script that arranges to start each UML and configure it appropriately
   for the test. It then starts listening (using uml_netjig) to the
   simulated network answering ARPs and inserting packets as appropriate.

   umlXtest has a series of slots, each of which should be filled by a
   host. The list of slots is controlled by the variable, XHOST_LIST.
   This variable should be set to a space separated list of slots. The
   former umlplutotest is now implemented as a variation of the umlXhost
   test, with XHOST_LIST="EAST WEST".

   For each host slot that is defined, a series of variables should be
   filled in, defining what configuration scripts to use for that host.

   The following are used to control the console input and output to the
   system. Where the string ${host} is present, the host slot should be
   filled in. I.e. for the two host system with XHOST_LIST="EAST WEST",
   then the variables: EAST_INIT_SCRIPT and WEST_INIT_SCRIPT will exist.

   ${host}HOST
          The name of the UML host which will fill this slot

   ${host}_INIT_SCRIPT
          a file of commands that is fed into the virtual machine's
          console in single user mode prior to starting the tests. This
          file will usually set up any eroute's and SADB entries that are
          required for the test. Similar to INIT_SCRIPT, above.

   ${host}_RUN_SCRIPT
          a file of commands that is fed into the virtual machine's
          console in single user mode, before the packets are sent. This
          set of commands is run after all of the virtual machines are
          initialized. I.e. after EAST_INIT_SCRIPT AND WEST_INIT_SCRIPT.
          This script can therefore do things that require that all
          machines are properly configured.

   ${host}_RUN2_SCRIPT
          a file of commands that is fed into the virtual machine's
          console in single user mode, after the packets are sent. This
          set of commands is run before any of the virtual machines have
          been shut down. (I.e. before EAST_FINAL_SCRIPT AND
          WEST_FINAL_SCRIPT.) This script can therefore catch
          post-activity status reports.

   ${host}_FINAL_SCRIPT
          a file of commands that is fed into the virtual machine's
          console in single user mode after the final packet is sent.
          Similar to INIT_SCRIPT, above. If not specified, then the
          single command "halt" is sent. Note that when this script is
          run, the other virtual machines may already have been killed.
          If specified, then the script should end with a halt command to
          nicely shutdown the UML.

   REF_${host}_CONSOLE_OUTPUT
          Similar to REF_CONSOLE_OUTPUT, above.

   Some additional flags apply to all hosts:

   REF_CONSOLE_FIXUPS
          a list of scripts (found in klips/test/fixups) to apply to
          sanitize the console output of the machine under test. These
          are typically perl, awk or sed scripts that remove things in
          the kernel output that change each time the test is run and/or
          compiled.

   In addition to input to the console, the networks may have input fed
   to them:

   EAST_INPUT/WEST_INPUT
          a [10]pcap file to feed in on the private network side of each
          network. The "EAST" and "WEST" here refer to the networks, not
          the hosts.

   REF_PUB_FILTER
          a program that will filter the TCPDUMP output to do further
          processing. Defaults to "cat".

   REF_EAST_FILTER/REF_WEST_FILTER
          a program that will filter the TCPDUMP output to do further
          processing. Defaults to "cat".<

   TCPDUMPFLAGS
          a set of flags for the tcpdump used when converting captured
          output. Typical values will include "-n" to turn off DNS, and
          often "-E" to set the decryption key (tcpdump 3.7.1 and higher
          only) for ESP packets. The "-t" flag (turn off timestamps) is
          provided automatically

   REF_EAST_OUTPUT/REF_WEST_OUTPUT
          a text file containing tcpdump output. Packets on the private
          (eth0) interface are captured and compared after conversion by
          tcpdump, as with REF_PUB_OUTPUT.

          There are two additional environment variables that may be set
          on the command line:

        NETJIGVERBOSE=verbose export NETJIGVERBOSE
                If set, then the test output will be "chatty", and let
                you know what commands it is running, and as packets are
                sent. Without it set, the output is limited to
                success/failure messages.

        NETJIGTESTDEBUG=netjig export NETJIGTESTDEBUG
                This will enable debugging of the communication with
                uml_netjig, and turn on debugging in this utility. This
                does not imply NETJIGVERBOSE.

   HOSTTESTDEBUG=hosttest export HOSTTESTDEBUG
          This will show all interactions with the user-mode-linux
          consoles

kernel_patch_test paramaters

   The kernel_patch_test function takes some kernel source, copies it
   with lndir, and then applies the patch as produced by "make
   kernelpatch".

   The following are used to control the input and output to the system:

   KERNEL_NAME
          the kernel name, typically something like "linus" or "rh"

   KERNEL_VERSION
          the kernel version number, as in "2.2" or "2.4".

   KERNEL_${KERNEL_NAME}${KERNEL_VERSION}_SRC
          This variable should set in the environment, probably in
          ~/freeswan-regress-env.sh. Examples of this variables would be
          KERNEL_LINUS2_0_SRC or KERNEL_RH7_3_SRC. This variable should
          point to an extracted copy of the kernel source in question.

   REF_PATCH_OUTPUT
          a copy of the patch output to compare against

   KERNEL_PATCH_LEAVE_SOURCE
          If set to a non-empty string, then the patched kernel source is
          not removed at the end of the test. This will typically be set
          in the environment while debugging.

module_compile paramaters

   The module_compile test attempts to build the KLIPS module against a
   given set of kernel source. This is also done by the RPM tests, but in
   a very specific manner.

   There are two variations of this test - one where the kernel either
   doesn't need to be configured, or is already done, and tests were
   there is a local configuration file.

   Where the kernel doesn't need to be configured, the kernel source that
   is found is simply used. It may be a RedHat-style kernel, where one
   can cause it to configure itself via rhconfig.h-style definitions. Or,
   it may just be a kernel tree that has been configured.

   If the variable KERNEL_CONFIG_FILE is set, then a new directory is
   created for the kernel source. It is populated with lndir(1). The
   referenced file is then copied in as .config, and "make oldconfig" is
   used to configure the kernel. This resulting kernel is then used as
   the reference source.

   In all cases, the kernel source is found the same was for the
   kernelpatch test, i.e. via KERNEL_VERSION/KERNEL_NAME and
   KERNEL_${KERNEL_NAME}${KERNEL_VERSION}_SRC.

   Once there is kernel source, the module is compiled using the
   top-level "make module" target.

   The test is considered successful if an executable is found in
   OUTPUT/module/ipsec.o at the end of the test.

   KERNEL_NAME
          the kernel name, typically something like "linus" or "rh"

   KERNEL_VERSION
          the kernel version number, as in "2.2" or "2.4".

   KERNEL_${KERNEL_NAME}${KERNEL_VERSION}_SRC
          This variable should set in the environment, probably in
          ~/freeswan-regress-env.sh. Examples of this variables would be
          KERNEL_LINUS2_0_SRC or KERNEL_RH7_3_SRC. This variable should
          point to an extracted copy of the kernel source in question.

   KERNEL_CONFIG_FILE
          The configuration file for the kernel.

   KERNEL_PATCH_LEAVE_SOURCE
          If set to a non-empty string, then the configured kernel source
          is not removed at the end of the test. This will typically be
          set in the environment while debugging.

   MODULE_DEF_INCLUDE
          The include file that will be used to configure the KLIPS
          module, and possibly the kernel source.

                               Current pitfalls

   "tcpdump dissector" not available.
          This is a non-fatal warning. If uml_netjig is invoked with the
          -t option, then it will attempt to use tcpdump's dissector to
          decode each packet that it processes. The dissector is
          presently not available, so this option it normally turned off
          at compile time. The dissector library will be released with
          tcpdump version 4.0.

References

   1. http://user-mode-linux.sourceforge.net/
   2. file://localhost/syssrc/openswan-2.4-branch/doc/src/umltesting.html
   3. file://localhost/syssrc/openswan-2.4-branch/doc/src/nightly.html
   4. file://localhost/syssrc/openswan-2.4-branch/doc/src/makecheck.html#umlplutotest
   5. file://localhost/syssrc/openswan-2.4-branch/doc/src/makecheck.html#umlXhost
   6. http://expect.nist.gov/
   7. http://www.tcpdump.org/
   8. http://www.tcpdump.org/
   9. http://expect.nist.gov/
  10. http://www.tcpdump.org/
