Description of nc_test

# Description of `nc_test`

## Harvey Davies

### 7 August 1996

### TERMINOLOGY

Test
  Process of testing one netCDF library function.
Comparison
  Successful match of value of individual variable/attribute element read (possibly after put) with that expected.
Error
  Non-zero status returned by netCDF library function.
Failure
  Unexpected result (including error status). Possible causes:
  - absent data file
  - wrong permissions
  - bug in netCDF library function
Blunder
  Bug in `nc_test`.

### PROGRAM OPERATION AND COMMAND-LINE OPTIONS

All printed output is written to `stderr`, not `stdout`. This is intended to facilitate debugging.

There is a separate test for each netCDF library function (apart from `test_nc_enddef` which simply calls `test_nc_redef`, which tests both `nc_redef` and `nc_enddef`). Each test generates one line of results if there are no failures and additional lines if there are failures. The simplest such line for no failures is one such as:

    *** Testing nc_open ... ok

Many tests count the number of comparisons and if there are no failures print a line such as:

    *** Testing nc_get_var1_short ...  682 good comparisons. ok

Each failure produces a line until the limit (per test) specified by the `-n` option (default 8) is reached. E.g. running the full test in a read-only directory produces:

    *** Testing nc_open ...
            FAILURE at line 119 of test_read.c: nc_create: Permission denied
            FAILURE at line 124 of test_read.c: nc_open: No such file or directory
            FAILURE at line 129 of test_read.c: remove of scratch.nc failed
            ### 3 FAILURES TESTING nc_open! ###

The total number of failures is printed at the end. E.g.

    Total number of failures: 60

If there are no failures then `nc_test` exits with exit-status 0, otherwise 1.

The following usage instructions are produced by the command `nc_test -h`:

    nc_test [-hrv] [-n ]
       [-h] Print help
       [-c] Create file test.nc (Do not do tests)
       [-r] Just do read-only tests
       [-v] Verbose mode
       [-n ] max. number of messages per test (Default: 8)

The `-c` option creates the read-only test file test.nc and does nothing else. (Originally I had a separate program, but it was simpler to combine them.) The utility `make` will run `nc_test -c` to recreate `test.nc` whenever `nc_test` is changed, but I suggest including `test.nc` in the distribution so users do not normally have to do this step.

As mentioned above, it is possible to run the full test in a read-only directory. (This currently generates 60 failures.) But in a read-only directory it is better to use the `-r` option which suppresses all actions which write, so there should be no failures.

The option `-v` sets verbose mode, in which additional details are produced after certain failures. (This is not consistent across tests. An earlier version also produced details of each comparison, whether successful or not; but this proved too voluminous.)

The option `-n MAX_NMPT` sets the maximum number of messages printed per test. This defaults to 8.

### SOURCE FILES

`nc_test.c`
  Driver
`tests.h`
  Main include file
`error.c`
  Functions concerned with low-level printing and failure handling.
`error.h`
  Include file related to error.c.
`util.c`
  Various utility functions for generating data, comparisons, etc.
`test_get.m4`
  Read-only type-specific tests.
`test_put.m4`
  Write type-specific tests.
`test_read.c`
  Other read-only tests (in fact there is some writing).
`test_write.c`
  Other write tests.

The `wc` utility gives the following counts of lines, words and characters:

            22        70       622 error.h
           324      1000      9226 tests.h
           366      1303     11029 test_get.m4
           610      1941     16469 test_put.m4
            52       155      1159 error.c
           277       692      7645 nc_test.c
          1599      5908     45979 test_read.c
          1882      6887     55625 test_write.c
           728      2623     19434 util.c
          5860     20579    167188 total

### DATA FILES

Two netCDF files are used. One (`test.nc`) is read-only, except that the `-c` option creates it. The other (`scratch.nc`) has a lifetime within the period of any test which does writing.

The file `tests.h` is used merely as an example of a non-netCDF file.

### INTERNAL METADATA DESCRIBING TEST DATA

The test metadata defining the file `tests.nc` (and for many tests also `scratch.nc`) is described by parameters (e.g. `NVARS`, the number of variables) defined in `tests.h` and global variables with the prefix `dim`, `var`, `att` or `gatt` (global attribute). E.g. a netCDF variable is described by global variables such as `var_name`, `var_type` and `var_dimid`. Note that the data values of netCDF variables and attributes are not stored in this way. Instead, these are generated by function `hash` each time they are needed.

The need for separate global attribute tests is obviated by accessing `att/gatt` variables via macros which give global values for a varid of -1. E.g. `ATT_NAME(varid,j)` gives `gatt_name[j]` if varid is -1, otherwise it gives `att_name[varid][j]`. Thus attribute tests will typically vary `varid` from -1 to `NVARS-1`.

The function `init_gvars` (in `util.c`) defines the values of these global variables defining the metadata. It defines variable names such as `sr23` for a 3D short dimensioned `UNLIMITED` (record) x 2 x 3. The maximum dimension size is 4. There is a vector of length 1, 2, 3 and 4 for each data type. However for ranks 2 and 3 (`MAX_RANK`) there is only one type for each shape, but there is an example of each possible shape (including one with the `UNLIMITED` dimension). This produces a total of 136 (`NVARS`) variables.

The command

    ncdump -h test.nc | grep char

produces the following list of all the character variables:

            char c ;
            char cr(Dr) ;
            char c1(D1) ;
            char c2(D2) ;
            char c3(D3) ;
            char c4(D4) ;
            char cr1(Dr, D1) ;
            char c13(D1, D3) ;
            char c31(D3, D1) ;
            char c43(D4, D3) ;
            char cr21(Dr, D2, D1) ;
            char cr33(Dr, D3, D3) ;
            char c111(D1, D1, D1) ;
            char c123(D1, D2, D3) ;
            char c141(D1, D4, D1) ;
            char c213(D2, D1, D3) ;
            char c231(D2, D3, D1) ;
            char c243(D2, D4, D3) ;
            char c321(D3, D2, D1) ;
            char c333(D3, D3, D3) ;
            char c411(D4, D1, D1) ;
            char c423(D4, D2, D3) ;
            char c441(D4, D4, D1) ;

Given that there are two records (`Dr` = 2 currently), one can use this to calculate the number of character elements as: 1+2+1+2+3+4+2+3+3+12+4+18+1+6+4+6+6+24+6+27+4+24+16 = 179. This corresponds to the number of comparisons for characters given in output such as:

    *** Testing nc_get_vara_text ...  179 good comparisons. ok

The number of comparisons for numeric types is more difficult to calculate because it depends on whether the data values are within the ranges for the external and internal types. However one can do quite a number of consistency checks. For example the number of comparisons done by the `(void *)` functions (e.g. `nc_get_var1`) is 1386 which equals the sum of that for `nc_get_var1_double` (1207, the total number of numeric elements) and the above value of 179.

There is one global attribute of each data-type and lengths vary from 1 to 6. Only the six scalar variables have attributes and their lengths vary from 0 to 5.

### GENERATION OF TEST DATA VALUES

Variable and attribute data values are generated by function `hash` in file `util.c`. The type `double` result is a function of the data-type, rank (-1 for attribute) and index vector.

Vectors are treated specially. Index 0 gives the minimum for the type and index 1 gives the maximum. For types other than `NC_CHAR` and `NC_DOUBLE`, index 2 gives a value just less than the minimum and index 3 gives a value just more than the maximum.

Otherwise `hash` uses an algorithm which always generates a valid value for the type.

There is also a 4-argument wrapper function `hash4` which has an additional argument specifying whether the internal type is `uchar`. This handles the special `NC_BYTE/uchar` adjustment.

### ncdump of `test.nc`

The following is the start of output from `ncdump test.nc`:

    netcdf test {
    dimensions:
    	Dr = UNLIMITED ; // (2 currently)
    	D1 = 1 ;
    	D2 = 2 ;
    	D3 = 3 ;
    	D4 = 4 ;
    variables:
    	char c ;
    	byte b ;
    	    b:c = "" ;
    	short s ;
    	    s:b = '\200' ;
    	    s:s = -32768s, 32767s ;
    	long l ;
    	    l:l = -2147483648, 2147483647, 2147483647 ;
    	    l:f = -3.4028235e+38f, 3.4028235e+38f, -3.4028235e+38f, 3.4028235e+38f ;
    	    l:d = -1.797693134862316e+308, 1.797693134862316e+308, -1., 1., 660. ;
    	float f ;
    	double d ;
    	    d:c = "˙AZ$&" ;
    	char cr(Dr) ;
    	byte br(Dr) ;
    	short sr(Dr) ;
    	long lr(Dr) ;
    	float fr(Dr) ;
    	double dr(Dr) ;
    	char c1(D1) ;
    	byte b1(D1) ;
    	short s1(D1) ;
    	long l1(D1) ;
    	float f1(D1) ;
    	double d1(D1) ;
    	char c2(D2) ;
    	byte b2(D2) ;
    	short s2(D2) ;
    	long l2(D2) ;
    	float f2(D2) ;
    	double d2(D2) ;
    	char c3(D3) ;
    	byte b3(D3) ;
    	short s3(D3) ;
    	long l3(D3) ;
    	float f3(D3) ;
    	double d3(D3) ;
    	char c4(D4) ;
    	byte b4(D4) ;
    	short s4(D4) ;
    	long l4(D4) ;
    	float f4(D4) ;
    	double d4(D4) ;
    	char cr1(Dr, D1) ;
    	byte br2(Dr, D2) ;
    	short sr3(Dr, D3) ;
    	long lr4(Dr, D4) ;
    	float f11(D1, D1) ;
    	double d12(D1, D2) ;
    	char c13(D1, D3) ;
    	byte b14(D1, D4) ;
    	short s21(D2, D1) ;
    	long l22(D2, D2) ;
    	float f23(D2, D3) ;
    	double d24(D2, D4) ;
    	char c31(D3, D1) ;
    	byte b32(D3, D2) ;
    	short s33(D3, D3) ;
    	long l34(D3, D4) ;
    	float f41(D4, D1) ;
    	double d42(D4, D2) ;
    	char c43(D4, D3) ;
    	byte b44(D4, D4) ;
    	short sr11(Dr, D1, D1) ;
    	long lr12(Dr, D1, D2) ;
    	float fr13(Dr, D1, D3) ;
    	double dr14(Dr, D1, D4) ;
    	char cr21(Dr, D2, D1) ;
    	byte br22(Dr, D2, D2) ;
    	short sr23(Dr, D2, D3) ;
    	long lr24(Dr, D2, D4) ;
    	float fr31(Dr, D3, D1) ;
    	double dr32(Dr, D3, D2) ;
    	char cr33(Dr, D3, D3) ;
    	byte br34(Dr, D3, D4) ;
    	short sr41(Dr, D4, D1) ;
    	long lr42(Dr, D4, D2) ;
    	float fr43(Dr, D4, D3) ;
    	double dr44(Dr, D4, D4) ;
    	char c111(D1, D1, D1) ;
    	byte b112(D1, D1, D2) ;
    	short s113(D1, D1, D3) ;
    	long l114(D1, D1, D4) ;
    	float f121(D1, D2, D1) ;
    	double d122(D1, D2, D2) ;
    	char c123(D1, D2, D3) ;
    	byte b124(D1, D2, D4) ;
    	short s131(D1, D3, D1) ;
    	long l132(D1, D3, D2) ;
    	float f133(D1, D3, D3) ;
    	double d134(D1, D3, D4) ;
    	char c141(D1, D4, D1) ;
    	byte b142(D1, D4, D2) ;
    	short s143(D1, D4, D3) ;
    	long l144(D1, D4, D4) ;
    	float f211(D2, D1, D1) ;
    	double d212(D2, D1, D2) ;
    	char c213(D2, D1, D3) ;
    	byte b214(D2, D1, D4) ;
    	short s221(D2, D2, D1) ;
    	long l222(D2, D2, D2) ;
    	float f223(D2, D2, D3) ;
    	double d224(D2, D2, D4) ;
    	char c231(D2, D3, D1) ;
    	byte b232(D2, D3, D2) ;
    	short s233(D2, D3, D3) ;
    	long l234(D2, D3, D4) ;
    	float f241(D2, D4, D1) ;
    	double d242(D2, D4, D2) ;
    	char c243(D2, D4, D3) ;
    	byte b244(D2, D4, D4) ;
    	short s311(D3, D1, D1) ;
    	long l312(D3, D1, D2) ;
    	float f313(D3, D1, D3) ;
    	double d314(D3, D1, D4) ;
    	char c321(D3, D2, D1) ;
    	byte b322(D3, D2, D2) ;
    	short s323(D3, D2, D3) ;
    	long l324(D3, D2, D4) ;
    	float f331(D3, D3, D1) ;
    	double d332(D3, D3, D2) ;
    	char c333(D3, D3, D3) ;
    	byte b334(D3, D3, D4) ;
    	short s341(D3, D4, D1) ;
    	long l342(D3, D4, D2) ;
    	float f343(D3, D4, D3) ;
    	double d344(D3, D4, D4) ;
    	char c411(D4, D1, D1) ;
    	byte b412(D4, D1, D2) ;
    	short s413(D4, D1, D3) ;
    	long l414(D4, D1, D4) ;
    	float f421(D4, D2, D1) ;
    	double d422(D4, D2, D2) ;
    	char c423(D4, D2, D3) ;
    	byte b424(D4, D2, D4) ;
    	short s431(D4, D3, D1) ;
    	long l432(D4, D3, D2) ;
    	float f433(D4, D3, D3) ;
    	double d434(D4, D3, D4) ;
    	char c441(D4, D4, D1) ;
    	byte b442(D4, D4, D2) ;
    	short s443(D4, D4, D3) ;
    	long l444(D4, D4, D4) ;

    // global attributes:
    :Gc = "" ;
    :Gb = '\200', '\177' ;
    :Gs = -32768s, 32767s, 32767s ;
    :Gl = -2147483648, 2147483647, 2147483647, 2147483647 ;
    :Gf = -3.4028235e+38f, 3.4028235e+38f, -3.4028235e+38f, 3.4028235e+38f, 531.f ;
    :Gd = -1.797693134862316e+308, 1.797693134862316e+308, -1., 1., 660., 650. ;

    data:
     c = "\002" ;
     b = 254 ;
     s = -5 ;
     l = -20 ;
     f = -9  ;
     d = -10 ;
     cr = "\000\377" ;
     br = 128, 127 ;
     sr = -32768, 32767 ;
     lr = -2147483648, 2147483647 ;
     fr = -3.402823e+38 , 3.402823e+38  ;
     dr = -1.79769313486232e+308, 1.79769313486232e+308 ;
     c1 = "" ;
     b1 = 128 ;
     s1 = -32768 ;

### HANDLING OF FAILURES

The macro `IF` is used in place of an ordinary `if` as in:

        IF (err)
            error("nc_open: %s", nc_strerror(err));

`IF` acts like an ordinary `if` except for two side effects. Firstly, it prints a failure message. Secondly, it increments the global variable `nfails`, which contains a count of the failures in the current test. The true condition should always correspond to a failure. There can be an `else`.

The macro `IF` is defined in file `error.h` as follows:

    #define IF(EXPR) if (ifFail(EXPR, __LINE__, __FILE__))

The function `ifFail` is defined in file `error.c` as follows:

    int
    ifFail(const int expr, const int line, const char *file)
    {
        if (expr) {
            ++nfails;
            error("\n\tFAILURE at line %d of %s: ", line, file);
        }
        return expr;
    }

The driver calls each test function and prints summary messages using the following macro `NC_TEST` defined in file `nc_test.c`:

    #define NC_TEST(func) \
        print( "*** Testing " #func " ... ");\
        nfails = 0;\
        test_ ## func();\
        nfailsTotal += nfails;\
        if (verbose) \
            print("\n"); \
        if ( nfails == 0) \
            print( "ok\n");\
        else\
            print( "\n\t### %d FAILURES TESTING %s! ###\n", nfails, #func)

### INDIVIDUAL TESTS

The file `scratch.nc` is created at the start, and deleted at the end, of each test that uses it.

An attempt has been made to generate all possible errors (including all possible illegal argument values) and check their error status values. Where some arguments could be `NULL`, this `NULL` alternative is typically chosen for odd variable ID values.

In most cases the get/put tests read/write each variable/attribute element exactly once.

The `get_vara/put_vara` tests split each dimension at a randomly selected point. So a matrix is divided into four sub-matrices.

The `get_vars/put_vars` tests do the same splitting, but also randomly select a stride vector for each sub-array. The whole sub-array is covered by repeating with different start-index-vectors. (The number of different starts is the product of the elements of the stride vector.)

The buffer for get/put att, `var1`, `vara` and `vars` is used for only a single get/put. However the buffer for `varm` tests is a bit-image of the external variable and each get/put transfers some elements, filling in the gaps. Otherwise, these `varm` tests are similar to the `vars` tests.

### TESTING DONE

Tests were run on slim, binnie and laraine. These tests consisted of:

    nc_test -c  # in writable directory
    nc_test     # in writable directory
    nc_test -r  # in read-only directory
    nc_test     # in read-only directory

A full test on slim gave the following timings:

    real    0m17.67s
    user    0m7.81s
    sys     0m1.15s

### CURRENT STATUS AND WORK REMAINING

Currently there are 95 tests implemented. There are 32 tests not yet implemented, corresponding to library functions have not yet been implemented. These correspond to the eight types for each of `nc_get_vars_`TYPE, `nc_get_varm_`TYPE, `nc_put_vars_`TYPE, `nc_put_varm_`TYPE. However these will all be generated by four `m4` macros, which will be similar to the existing `nc_get_vars`, `nc_get_varm`, `nc_put_vars`, `nc_put_varm`.

The code to test negative strides is currently commented out, because this feature has not yet been implemented in the library.