Parallel NetCDF Release Notes:
=====================================

-------------------------------------
version  1.7.0 (March 3, 2016)
-------------------------------------

  o New features
    * Nonblocking requests now can be posted while the program is in the define
      mode. This feature enables applications to add new variables and post the
      nonblocking write requests without separating the codes by
      ncmpi_enddef(). Note ncmpi_wait_all and ncmpi_wait must still be called
      in the data mode.
    * When using immutable write buffer in put APIs on a Little Endian machine,
      the in-place byte swap operation performed internally in PnetCDF can
      cause a fatal error (by trying to change the contents of an immutable
      memory space). The solution of copying user's write request to a
      temporary buffer and perform byte swap there conflicts with PnetCDF's
      design principle of avoiding internal memory allocation as much as
      possible. Since the size of immutable buffer in most cases is small,
      PnetCDF now compromise the two by making a copy of write requests that is
      less than 4KB. Users are warned that using immutable write buffer larger
      than 4KB will still cause the fatal error in the PnetCDF default
      configuration. The way to completely disable in-place byte swap is to
      build PnetCDF with --disable-in-place-swap option at the configure time.
      In this case, the internal memory allocation in PnetCDF will increase.

  o Syntax changes
    * In ncmpi_wait_all(), ncmpi_wait(), and ncmpi_cancel(), the API's third
      argument, array of requests IDs, is changed to INOUT. Upon successful
      completion or cancellation of individual nonblocking requests, the
      corresponding request IDs are set to NC_REQ_NULL.

  o New run-time environment variables
    * PNETCDF_VERBOSE_DEBUG_MODE environment variable can be used to print the
      location in the source code where the error code is originated, no matter
      the error is intended or not. This run-time environment variable only
      takes effect when PnetCDF is configure with debug mode, i.e.
      --enable-debug is used at the configure command line.  Set this variable
      to 1 to enable. Set it to 0 or keep it unset disables this mode. Default
      is 0, i.e. disabled.  Users are warned that enabling this mode may result
      in a lot of debugging messages printed in stderr.

  o New example programs
    * examples/C/nonblocking_write_in_def.c shows an example of posting
      nonblocking write requests in the define mode.
    * examples/C/req_all.c shows an example of using NC_REQ_ALL to flush all
      pending nonblocking requests without providing the requests IDs.

  o New test program
    * test/cdf_format/dim_cdf12.c tests defining maximal dimension size for
      CDF-1 and CDF-2 file formats
    * test/testcases/test_erange.c tests if the error code NC_ERANGE can be
      correctly returned for two cases 1) get a NC_UBYTE value of 255 from a
      netCDF file to a memory buffer of type signed char and 2) put a value
      of -1 of signed char to a NC_UBYTE variable in a netCDF file
    * test/testcases/check_type.c tests if the error codes can be correctly
      returned when conflicted in-memory and extenal data types are used.
    * test/testcases/put_parameter.f tests the use of immutable write buffer
      (e.g. a buffer declared as PARAMETER). Note the buffer size must be
      smaller than 4KB.
    * test/nonblocking/i_varn_indef.c tests posting nonblocking requests in
      define mode.
    * test/nonblocking/req_all.c tests the use of NC_REQ_ALL for flushing
      all pending nonblocking requests without providing the requests IDs.
    * test/last_large_var.c tests the special case when there is no record
      variable, the last fixed-size variable can be larger than 2GiB in size
      if its starting file offset is less than 2GiB.
    * test/testcases/buftype_free.c and test/testcases/buftype_freef.f test
      the bug in r2160.
    * testcases/add_var.c checks the starting file offsets of newly added
      variables from re-entering the define mode.
    * testcases/attrf.f checks NF_ERANGE is returned instead of coredump. This
      is particularly for NAG Fortran compiler that may report "Arithmetics
       exception".
    * testcases/check_striping.c checks if the file striping unit and factor
      returned from MPI-IO hints are consistent among processes.
    * test/nonblocking/column_wise.c checks if PnetCDF detects interleaved
      fileviews from multiple nonblocking requests and correctly breaks and
      reconstructs the fileviews so the combined fileview is monotonic
      non-decreasing in file offsets.

  o New optimization
    * Filling variables at ncmpi_enddef() is now done by aggregating all write
      requests into one MPI collective write call. In v 1.6.1, this is done by
      filling one variable at a time.

  o New utility program
    * ncoffsets reports the file offset information, including the starting and
      ending file offsets, of variables stored in a netCDF file. ncoffsets is
      compiled with gcc if gcc is presented on the build system. Additional
      command-line options are: (-v) reports only for a selected list of
      variables in interest, (-s) prints the variable sizes, (-g) outputs the
      file space gap size from the end of previous variable, (-x) reports
      whether there is a gap between any two adjacent fixed-size variables. See
      the man page for descriptions of all command-line options and examples.

  o Semantics updates
    * All nonblocking APIs now take a NULL pointer for the request ID argument,
      meaning users do not wish to keep track of the request ID. If NULL
      request IDs are used, NC_REQ_ALL should be used when calling
      ncmpi_wait_all/ncmpi_wait to commit all the pending nonblocking requests.
      This feature relinquishes users from the responsibility of tracking the
      IDs of pending requests.
    * Using NC_REQ_ALL as the 2nd argument "num" in ncmpi_wait_all/ncmpi_wait
      APIs will flush all the pending nonblocking requests. In this case, the
      3rd and 4th arguments "array_of_requests" and "array_of_statuses" will be
      ignored and thus these two arguments can be NULLs.
    * Using NC_REQ_ALL in ncmpi_cancel() will cancel all the pending
      nonblocking requests.
    * Using NC_GET_REQ_ALL or NC_PUT_REQ_ALL in ncmpi_wait_all(), ncmpi_wait(),
      and ncmpi_cancel() for all the pending get-only or put-only requests,
      respectively.

  o Other updates:
    * Conform with netCDF on the maximal dimension size for CDF-2 file format
      to be (2^32 - 4)
    * NC_ERANGE checks have been removed from nc_test for text APIs and
      variables that are defined as NC_CHAR type
    * Add README.K-Computer build recipe for using Fujitsu MPI compilers on the
      K computer at RIKEN in Japan
    * Add README.INTEL build recipe for using Intel MPI compilers 4.x
    * Build dependency rule is added for files configure and configure.in
    * PnetCDF checks MPICC/MPICXX/MPIF77/MPIF90 instead of CC/CXX/F77/F90/FC.
      If MPICC/MPICXX/MPIF77/MPIF90 are set, PnetCDF will ignore
      CC/CXX/F77/F90/FC. If CC/CXX/F77/F90/FC is set instead of
      MPICC/MPICXX/MPIF77/MPIF90, PnetCDF will now copy them to
      MPICC/MPICXX/MPIF77/MPIF90.
    * Enforce netCDF convention on error code priority: NC_ECHAR trumps
      NC_EINVALCOORDS, NC_EEDGE, and NC_ESTRIDE.
    * Return error code NC_EGLOBAL instead of NC_ENOTVAR for APIs where using
      NC_GLOBAL as the variable ID argument is prohibited.
    * All Fortran 77 test and example programs (files with .f and .F
      extensions) have been revised to conform with 77 standard, and was tested
      using pgf77.
    * Now provides a pkg-config file
      (http://www.freedesktop.org/wiki/Software/pkg-config/), making it
      slightly easier to set the correct pnetcdf include and library paths.

  o Bug fixes
    * Fix the bus error of invalid address alignment when build with Fujitsu
      compiler. See r2171 and r2180.
    * Fix the bug for the special case when there is no record variable, the
      last fixed-size variable can be larger than 2GiB in size if its starting
      file offset is less than 2GiB. See r2166.
    * Fix the nonblocking flexible APIs that fail to save (duplicate) the user
      MPI derived data type that later is needed to unpack read data to the
      user buffer (a call to MPI_Unpack). See r2160.
    * Fix Fortran 77 constants nf_fill_uint and nf_fill_int64 (thanks Jim
      Edwards) that pgf77-based MPI compiler does not like "_8" modifier.
      See r2045 and r2051. The same issue for NAG Fortran compiler is also
      resolved. See r2089 and r2093.
    * In the example program examples/tutorial/pnetcdf-write-nb.c, the write
      buffers used in two iput API calls should be different. See r2095.
    * Fix the error reporting mechanism for NC_ERANGE, for when an arithmetic
      overflow happens. Overflow checking is now performed before I/O.

-------------------------------------
version  1.6.1 (June 1, 2015)
-------------------------------------

  o New features
    * PnetCDF now supports fill mode. ncmpi_set_fill() can be used to set the
      fill mode for the entire data set. Note the differences from netCDF:
      1. The default mode in PnetCDF is NC_NOFILL. 
      2. Setting fill mode for the entire file or individual variables must be
         done in define mode.
      3. For non-record variables, they are filled at the time ncmpi_enddef()
         is called.
      4. For record variables, users must explicitly call ncmpi_fill_var_rec()
         to fill one record at a time before writing to the record of that
         variable.

  o New APIs
    * ncmpi_def_var_fill() sets fill mode for an individual variable. This API
      must be called in define mode.
    * ncmpi_inq_var_fill() inquires fill mode of a variable.
    * ncmpi_fill_var_rec() is a collective API that fills a record of a record
      variable. This API must be called at data mode.
    * ncmpi_inq_default_format() for inquiring the default file format for
      new file creation. Note the default format can be changed by
      ncmpi_set_default_format().
    * The above new API are also available in Fortran and C++ versions.

  o New error code
    * NC_ENOTRECVAR when attempting operation only for record variables
    * NC_ENOTFILL when attempting to fill a variable when its fill mode is off
    * NC_EMULTIDEFINE_FILL_MODE	when inconsistent dataset fill mode is detected
    * NC_EMULTIDEFINE_VAR_FILL_MODE when inconsistent variable fill mode is
      detected
    * NC_EMULTIDEFINE_VAR_FILL_VALUE when inconsistent variable fill value is
      detected
    * Fortran versions of the above error codes are also added.

  o New example programs
    * C/fill_mode.c shows the use of ncmpi_set_fill(), ncmpi_def_var_fill(),
      ncmpi_inq_var_fill() and ncmpi_fill_var_rec()
      F77/fill_mode.f is the Fortran version.
      F90/fill_mode.f90 is the Fortran 90 version.
      CXX/fill_mode.cpp is the C++ version.
    * C/ghost_cell.c shows how to use varm API for writing from a user buffer
      as a 2D array with ghost cells on both ends of every dimension.

  o New test programs
    * nc_test/tst_nofill.c borrowed from netCDF
    * testcases/ivarn.c tests bug fix r2023 when the request IDs stored in
      argument array_of_requests[] of ncmpi_wait_all() are not in an
      increasing order.

  o Other updates:
    * Change the chunk size used for moving variable data when the file header
      extent expands. The default is now 1MB. If the file's striping unit
      size is known (from MPI-IO hint striping_unit) then the chunk size is
      set to the striping unit size.

  o Bug fixes
    * Add missing F90 function overloading for f90mpi_put_var_all,
      f90mpi_get_var_all, f90mpi_put_vard_all, and f90mpi_get_vard_all,
      when the user buffer is a scalar.
    * Fix when the request IDs passed in argument array_of_requests[] of
      ncmpi_wait_all() are not in an increasing order. See r2023.
    * Fix C++ compile error for converting NULL to string. See r2039.

-------------------------------------
version  1.6.0 (February 2, 2015)
-------------------------------------

  o Format conformation updates:
    * Conform with netCDF4 on CDF-1 and CDF-2 formats. The only difference now
      between the two formats are the OFFSET item in the format spec (32 vs.
      64 bit integers.) All names (variable, dimension, attribute) now allow
      extended characters (e.g. special2 and MUTF8).

  o New APIs
    * Nonblocking buffered varn API family.
      For C,   ncmpi_bput_varn_<type>()
      For F77, nfmpi_bput_varn_<type>()
      For F90, nf90mpi_bput_varn()
      For C++, NcmpiVar::bputVarn()
    * Nonblocking varn API family.
      For C,   ncmpi_iput_varn_<type>() and ncmpi_iget_varn_<type>()
      For F77, nfmpi_iput_varn_<type>() and nfmpi_iget_varn_<type>()
      For F90, nf90mpi_iput_varn()      and nf90mpi_iget_varn()
      For C++, NcmpiVar::iputVarn()     and NcmpiVar::igetVarn()
    * Blocking vard API family takes an argument of MPI derived data type that
      describes the file access layout, as opposed to vara and vars APIs that
      use start[] and count[].
      For C,   ncmpi_put_vard()    and ncmpi_get_vard()
      For F77, nfmpi_put_vard()    and nfmpi_get_vard()
      For F90, nf90mpi_put_vard()  and nf90mpi_get_vard()
      For C++, NcmpiVar::putVard() and NcmpiVar::getVard()
    * Collective var1 API family
      For C,   ncmpi_put_var1_all()        ncmpi_get_var1_all()
               ncmpi_put_var1_<type>_all() ncmpi_get_var1_<type>_all()
      For F77, nfmpi_put_var1_all()        nfmpi_get_var1_all()
               nfmpi_put_var1_<type>_all() nfmpi_get_var1_<type>_all()
      For F90, nf90mpi_put_var_all()       nf90mpi_get_var_all()
      For C++, NcmpiVar::putVar_all()      NcmpiVar::getVar_all()
    * ncmpi_inq_buffer_size() returns the size of buffer previously attached
      for use of bput APIs. With ncmpi_inq_buffer_usage() one can calculate
      the space remaining for additional bput requests.
      For F77, nfmpi_inq_buffer_size()
      For F90, nf90mpi_inq_buffer_size()
      For C++, NcmpiFile::Inq_buffer_size()
    * ncmpi_inq_recsize() returns the size of record block, i.e. the sum of
      single records of all record variables.
      For F77, nfmpi_inq_recsize()
      For F90, nf90mpi_inq_recsize()
      For C++, NcmpiGroup::getRecSize()
    * ncmpi_inq_num_rec_vars() and ncmpi_inq_num_fix_vars() report the number
      of record and fixed-size variables, respectively.
      For F77, nfmpi_inq_num_rec_vars()     and nfmpi_inq_num_fix_vars()
      For F90, nf90mpi_inq_num_rec_vars()   and nf90mpi_inq_num_fix_vars()
      For C++, NcmpiGroup::getRecVarCount() and NcmpiGroup::getFixVarCount()

  o New PnetCDF hint
    * pnetcdf_subfiling -- it can be set in an MPI info object or in the
      environment variable PNETCDF_HINTS to enable/disable subfiling.
      The value is either "enable" or "disable".

  o PnetCDF hint priority
    * The alignment hints set in the environment variable "PNETCDF_HINTS" have
      the highest priority, which overwrite the alignment hints set in
      ncmpi__enddef(), which overwrite the alignment hints set in the MPI_Info
      object used in the call of ncmpi_create() and ncmpi_open().

  o New error code
    * NC_ESTRICTCDF2 for attempting CDF-5 operation on CDF-2 file. For
      example, define a variable of type NC_INT64 in a CDF-2 file.
    * NC_ETYPESIZE when filetype size is bigger than the variable size
    * NC_ETYPE_MISMATCH when the element type of filetype mismatches the
      variable type
    * NC_ETYPESIZE_MISMATCH when filetype size mismatches buffer type size
    * NC_ENULLSTART when argument start is a NULL pointer
    * NC_ENULLCOUNT when argument count is a NULL pointer
    * NC_EINVAL_CMODE when invalid file create mode is set, (e.g. cannot have
      both NC_64BIT_OFFSET & NC_64BIT_DATA. In PnetCDF 1.5.0 and earlier
      versions, if both flags were set, then NC_64BIT_DATA triumphs
      NC_64BIT_OFFSET.)

  o New example programs
    * C/bput_varn_uint.c and F77/bput_varn_int8.f show the use of
      nonblocking bput_varn APIs
    * C/i_varn_int64.c and F77/i_varn_real.f show the use of nonblocking
      iput_varn and iget_varn APIs
    * C/vard_int.c F77/vard_int.f F90/vard_int.f90 CXX/vard_int.cpp show the
      use of vard API to write/read record and fixed-size variables.
    * C/transpose.c shows the use of ncmpi_put_vara_int_all to write a 3D array
      that is dimensional-transposed from the one stored in memory. Six
      transposed 3D arrays are saved whose dimensions are organized as ZYX,
      ZXY, YZX, YXZ, XZY, and XYZ. The C++, Fortran 77, and Fortran 90
      versions are also provided.

  o New test program
    * nonblocking/wait_after_indep.c tests if ncmpi_end_indep_data() returns
      properly when nonblocking APIs are called in independent data mode and
      the wait call is made after exiting the independent mode.
    * nonblocking/flexible_bput.c tests flexible bput APIs that use
      noncontiguous buffer type, noncontiguous imap and require type conversion
    * testcases/flexible2.c tests flexible APIs that perform type conversion
    * testcases/flexible_varm.c tests flexible varm APIs
    * testcases/varn_contig.c tests the case when the fileview is actually a
      contiguous chunk. PnetCDF should be able to merge all.
    * nonblocking/bput_varn_uint.c tests nonblocking bput_varn APIs
    * nonblocking/i_varn_int64.c tests nonblocking iput_varn and iget_varn APIs
    * test/testcases/test_vard.c test/testcases/test_vardf.f
      test/testcases/test_vardf90.f90 test the new vard APIs.
    * test/testcases/inq_recsize.c tests if one can correctly inquire the
      size of record block from in a netCDF file. A similar program in F90,
      named inq_recsizef.f90, is also added.
    * In test/nc_test, the test programs below are borrowed from netCDF test
      programs: t_nc.c tst_misc.c tst_norm.c tst_small.c tst_names.c
      tst_atts3.c tst_atts.c
    * test/testcases/one_record.c tests the special case defined in CDF-1 and
      CDF-2 specification that "A special case: Where there is exactly one
      record variable, we drop the requirement that each record be four-byte
      aligned, so in this case there is no record padding."
    * test/testcases/modes.c tests if correct error codes are returned when
      various file create/open modes are used.
    * Under test/testcases, varn_int.c varn_intf.f varn_real.f90 test varn APIs
    * test/testcases/inq_num_vars.c test if one can correctly inquire the
      numbers of record and fixed-size variables defined in a netCDF file.
      A similar program in F90, named inq_num_varsf.f90, is also added.
    * test/nonblocking/interleaved.c tests a combination of interleaved
      file types. This is to test the bug fix in r1758.

  o New optimization
    * Prior to this release, PnetCDF wraps each MPI read/write call in put/get
      APIs with two MPI_File_set_view(). One is before the MPI read/write call
      to take advantage of MPI's fileview feature for accessing non-contiguous
      file locations. The other is after the MPI read/write call to make the
      whole file visible, as the root process may write to file header later
      in the data mode and it alone cannot make a call to MPI_File_set_view()
      because the function is collective.

      In this release, the second MPI_File_set_view() has been removed. The
      root process's fileview is changed to always keep the whole file header
      visible. Saving a collective call to MPI_File_set_view() is expected to
      improve some performance.

  o Semantics updates
    * Header consistency mechanism has been updated. See README.consistency for
      details.
    * The use of NC_SHARE is also revised. See README.consistency for details.

  o Other updates:
    * The subfiling feature is now disabled in ncmpidump and ncmpidiff until
      a bug fix to allow reading the master file with the number of processes
      that is smaller than the number of subfiles.
    * The attribute names reserved for subfiling feature are now changed to
      use a prefix of "_PnetCDF_SubFiling". The leading "_" underscore is a CDL
      convention.
    * The flexible APIs now allow argument buftype to be MPI_DATATYPE_NULL.
      In this case, argument bufcount is ignored and argument buf's data type
      must match the data type of variable defined in the file - no data
      conversion will be done. This extension makes the flexible APIs be
      able to correspond to the netCDF APIs whose names do not contain a data
      type, e.g. nc_put_vara().
    * Type conversion between NC_BYTE and unsigned char no longer checks for
      out of range error (NC_ERANGE). CDF file format specification make a
      special case for this kind of data type conversion. See: "Note on byte
      data" in the format specification.
    * Conform with the CDF-2 and CDF-5 formats that names are normalized
      according to Unicode NFC normalization rules during encoding as UTF-8
      for storing in the file header.
    * A new configure option, --enable-large-file-test, to enable testing
      I/O on large files and large variables. Note the testing programs will
      run very slowly.

  o Bug fixes
    * fix the bug for flexible get_varn API. When buftype is noncontiguous, the
      bug forgot to "unpacks" the temporary buffer that reads data from file to
      the user buffer using buftype.
    * fix the bug in blocking flexible get APIs when buftype is noncontiguous,
      swap is needed, type conversion is not. The bug sets a NULL pointer to
      the read buffer and passes it to MPI_File_read functions. See r1815.
    * fix the bug for the NetCDF special case when there is only one record
      variable and the record size is not four-byte aligned. In this case,
      NetCDF spec says the alignment must dropped (no padding). See r1791.
    * fix the bug in nonblocking APIs when requests are resorted into
      nonoverlapping groups and the first group of file types are interleaved,
      it was mistakenly identified as non-interleaved. See r1758.
    * fix the setting for PNETCDF_RELEASE_DATE in configure.in. Using read
      command together with IFS does not work as expected in bash 4.3.11. See
      http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2014-July/001586.html
      for further detailed discussion.

-------------------------------------
version  1.5.0 (July 8, 2014)
-------------------------------------

  o New features
    * A new configure option "--disable-in-place-swap" is added. It disables
      the byte-swap operations running in-place on the user's write buffers.
      The purpose of providing this option is to deal with the problem when a
      Fortran program uses a immutable buffer for put APIs, e.g. the buffer is
      declared as a PARAMETER, and in-place byte swap on this buffer causes
      segmentation fault. See discussion threads of
      http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2013-July/001498.html

      Impacts:
      1. It takes effect only on Little Endian machines.
      2. It only affects put/iput data APIs, but not attribute APIs.
      3. The INTENT of buffer arguments in all Fortran 90 put/iput APIs will be
         declared as "IN". Without this setting, the default is "INOUT".
      4. It has an impact on performance, as an extra internal temporary buffer
         will be allocated to copy data over from user's put buffer, so byte
         swap can be run on the temporary buffer.

      The default setting is to enable in-place byte swap. PnetCDF tries not to
      allocate additional memory space, due to performance concern. Users are
      discouraged to use Fortran PARAMETER buffers in put APIs.

    * A new configure option "--enable-debug" is added. It enables a memory
      allocation tracing mechanism internal in PnetCDF. In addition, it enables
      the PnetCDF safe mode. (Note that setting the environment variable
      PNETCDF_SAFE_MODE at the run time can still overwrite the safe mode.)
      Default debug mode is disabled. When enabled, a user program can call
      three new APIs below: ncmpi_inq_malloc_size, ncmpi_inq_malloc_max_size,
      and ncmpi_inq_malloc_list to obtain the size in bytes of current memory
      allocated internally. This feature uses a binary tree to manage all
      malloc buffers, e.g. tsearch() and tdelete().

    * Add three Fortran parameters for PnetCDF library version numbers:
      PNETCDF_VERSION_MAJOR, PNETCDF_VERSION_MINOR, and PNETCDF_VERSION_SUB.
      Similarly in C programs, these are defined in pnetcdf.h as constants.

  o New APIs
    * C++ APIs are now available. They are developed based on netCDF-4 C++
      library. However, users are encouraged to use C APIs, instead C++.

    * ncmpi_put_att, ncmpi_get_att, nfmpi_put_att, and nfmpi_get_att - these
      APIs correspond to nc_put_att, nc_get_att, nf_put_att, and nf_get_att.
      Note they are not the "flexible" APIs. Flexible APIs have an MPI derived
      datatype argument.

    * ncmpi__enddef, nfmpi__enddef, and nf90mpi_enddef - these APIs correspond
      to netCDF nc__enddef, nf__enddef, and nf90_enddef (with additional
      optional arguments).

    * ncmpi_inq_file_info - the naming for ncmpi_get_file_info may cause
      confusion, as "get" has a different meaning on PnetCDF. The correct
      name should use "inq". However, ncmpi_get_file_info is kept for backward
      compatibility.

    * ncmpi_inq_striping, nfmpi_inq_striping, nf90mpi_inq_striping report the
      file system striping settings of the opened file: striping size and
      striping count, if the underneath MPI-IO can find their values from the
      file system in use.

    * ncmpi_inq_malloc_size, ncmpi_inq_malloc_max_size, ncmpi_inq_malloc_list
      report the size in bytes of current memory allocated internally by
      PnetCDF. Similar APIs for Fortran 77 and 90 are also available. These
      APIs are enabled only when PnetCDF is configured with option
      --enable-debug. When this option is not enabled, calling these APIs will
      return the error code NC_ENOTENABLED. These APIs are usefully for
      debugging.

    * ncmpi_inq_files_opened reports the number of files that are currently
      opened. Similar API for Fortran 77 and 90 are also available. The API
      takes 2 arguments: number of files and array of file IDs. If the second
      argument, array of IDs, is not NULL, then it will filled with the netCDF
      dataset IDs. This API is useful for debugging.

  o Syntax changes
    * For all Fortran put/iput APIs, the INTENT of write buffer arguments is
      changed to "INOUT" on Little Endian machines, if option
      "--disable-in-place-swap" is not used at configuration. Otherwise, i.e.
      on Big Endian machines or PnetCDF is configured with
      "--disable-in-place-swap" on Little Endian machines, the INTENT is "IN".

  o New PnetCDF hint
    * nc_record_align_size - aligns the starting file offset of the record
      variable section. Note this is for the entire section, not individual
      records.

  o New error code
    * NC_ENOTENABLED indicates the API is available only when the corresponding
      feature is enabled. For example, nfmpi_inq_malloc_size() returns this
      error code when "--enable-debug" is not used at configure.

    * NC_EBAD_FILE corresponds to MPI error code MPI_ERR_BAD_FILE, meaning
      "Invalid file name (e.g., path name too long)."

    * NC_ENO_SPACE corresponds to MPI error code MPI_ERR_NO_SPACE, meaning
      "Not enough space."

    * NC_EQUOTA corresponds to MPI error code MPI_ERR_QUOTA, meaning
      "Quota exceeded."

    * NC_EMULTIDEFINE_FNC_ARGS corresponds to MPI error code MPI_ERR_NOT_SAME,
      meaning "inconsistent function arguments used in collective API."

  o New run-time environment variables
    * none

  o New example programs
    * Example programs now report if there is any PnetCDF internal malloc
      residues yet to be freed, if --enable-debug option is used at
      configure.

    * Under examples/C, three examples are added: create_open.c, get_vara.c,
      and global_attributes.c. File examples/README contains their
      descriptions.

    * Under examples/CXX, several example programs corresponding to those in
      examples/C are added.

  o New programs for I/O benchmarks
    * none

  o New test program
    * Many test programs now report if there is any PnetCDF internal malloc
      residues yet to be freed, if --enable-debug option is used at configure.

    * add tests for flexible APIs. The tests borrow from nc_test/test_write.c
      that tests nc_put_var1, nc_put_vara, nc_put_vars, and nc_put_varm.
      Similarly for get APIs.

    * testcases/record.c checks if the number of records is updated correctly.
      It writes to a variable's 2nd record followed by the 1st record. A call
      to ncmpi_inq_dim() or ncmpi_inq_dimlen() should report 2 records after
      the writes complete.

    * testcases/noclobber.c checks if error code NC_EEXIST can be returned
      correctly when NC_NOCLOBBER modes is used in ncmpi_create and in the
      meantime the file exists.

    * Some test programs are developed to run in parallel. The test mechanism
      for parallel runs is command "make ptest". Two environment variables
      can be used to set the MPI run command and output file directory:
      TEST_MPIRUN and TEST_OUTDIR. Their defaults are mpiexec and "." (current
      directory), respectively. For example,
          make ptest TEST_MPIRUN="aprun -n NP" TEST_OUTDIR=/scratch
      Note the keyword "NP" will be replaced by the different numbers of
      processes used to run the tests. The testing uses up to 8 MPI processes.

    * A sample PBS script file is provided to test "make ptest" on machines
      with a batch queue system: pbs.script. This example script can be
      submitted from the build root directory (where you run "make" command to
      build PnetCDF library).

    * For cross compile environment (and batch queue system), the environment
      variable TEST_SEQRUN can be used to set the MPI run command. For example,
      the command for testing sequential programs:
          make check TEST_SEQRUN="aprun -n 1" TEST_OUTDIR=/scratch

      For non-cross compile environment, there is no need to set the environment
      variables, as long as one can run the MPI executable without mpirun or
      mpiexec.

  o New optimization
    * none

  o New utility program
    * pnetcdf_version prints the version information of the PnetCDF library and
      command-line arguments used at configure

  o Other updates:
    * Revise FLASH-IO benchmark to use nonblocking APIs for both checkpoint and
      plot writes. The control variable to switch between nonblocking and
      blocking API is "use_nonblocking_io". Set it to .FALSE. in
      flash_benchmark_io.F90 to switch to blocking APIs. Using nonblocking APIs
      is now the default.

    * To match ncdump, ncmpidump now only supports one input file.

    * Makefiles are revised for better recursive make and fixed some build
      target dependency for parallel make.

    * File creation was revised for handling NC_CLOBBER and NC_NOCLOBBER modes.
      On systems where Unix calls access() and unlink() are available, they
      are used to check if a file exits and to delete an existing file.

    * subfiling is now enabled by default. Users can use --disable-subfiling to
      disable it

    * man page of ncmpigen is updated to add the description for option "-v"
      which lets users to specify the desired output file formats, e.g.
      CDF-1, CDF-2, or CDF-5.

    * flex, lex, bison, or yacc are no longer needed for building ncmpigen.

  o Bug fixes
    * fix ncmpigen.y on parsing CDL file to get the number of records. The bug
      failed the command "make b-test", due to getting a wrong number of
      records (current value for the unlimited dimension).

    * fix the update for number of records when writing to a record that is
      not the last record.

-------------------------------------
version  1.4.1 (December 23, 2013)
-------------------------------------

  o Bug fix:
    * Improve pnetcdf.inc portability for fixed/free-form Fortran programs

  o Fortran API syntax changes
    * For nfmpi_put_att and nf90mpi_put_att family, the intent modifier for
      the put buffer arguments are now declared as INTENT(IN).
    * For nfmpi_put_var* and nf90mpi_put_var family
      + On Big Endian machines, the the intent modifier for the put buffer
        arguments are now declared as INTENT(IN).
      + On Little Endian machines, the the intent modifier for the put buffer
        arguments are still declared as INTENT(INOUT). This is because PnetCDF
        does in-place byte-swap on user's put buffer. If user's buffer is
        declared as parameter, then segment fault can happen when PnetCDF
        tries to byte-swap a read-only memory.

  o Subfiling
    * Subfiling is a new PnetCDF feature that divides a file transparently
      into several smaller subfiles, each of which stores subarrays
      in CDF file formats. The file name supplied by the users serves as a
      master file that contains all metadata about array partitioning
      information among the subfiles. Because data partitioning is made
      transparently from users, data accessing is kept the same through
      the conventional PnetCDF APIs and the master file.
    * To enable this feature at configure time, add configure option
      "--enable-subfiling".


-------------------------------------
version  1.4.0 (November 17, 2013)
-------------------------------------

  o New APIs
    * Fortran 90 APIs (adopted from netcdf-fortran-4.2). All F90 APIs have
      prefix name "nf90mpi_". The APIs support function overloading.
    * get/put_varn_<type> for reading/writing a list of subrequests (each is
      specified by starts[i][ndims] and counts[i][ndims] for subrequest i.
      ndims is the number of dimension of the variable) to a single variable.
    * multiple put/get requests with explicit buffer type names:
      ncmpi_mput_var_type(), ncmpi_mput_var1_type(), ncmpi_mput_vara_type(),
      ncmpi_mput_vars_type(), ncmpi_mput_varm_type(). Similar for get APIs.
      "type" is one of the followings: text, schar, uchar, short, ushort, int,
      uint, long, float, double, longlong, or ulonglong.
    * ncmpi_inq_nreqs() reports the number of pending nonblocking requests
    * ncmpi_inq_header_size() reports the size of the file header
    * ncmpi_inq_header_extent() reports the space currently allocated for the
      file header, (also the file offset of the first variable)
    * ncmpi_inq_put_size() reports the write amount committed by far
    * ncmpi_inq_get_size() reports the read amount committed by far
    * ncmpi_sync_numrecs() a collective API that can be called in independent
      data mode to synchronize the number of records in memory across all
      processes, and update to the file if NC_SHARE is set.

  o Syntax change for Fortran put APIs
    * intent of buffer argument in all Fortran APIs is changed to inout, as
      byte-swap might be performed directly on the buffer. This change is
      for performance consideration. For example, if the buffer is declared
      as Fortran parameter, then compile will fail.

  o New PnetCDF hint
    * nc_header_read_chunk_size: PnetCDF reads the file headers in chunks. This
      hint indicates the chunk size (in bytes). The default is 256 KB.

  o New error code
    * NC_EINTOVERFLOW reports the error of 4-byte integer overflow. This
      usually happens due to MPI-IO data type constructor APIs' arguments using
      4-byte integers.
    * Error codes to report metadata defined inconsistently across processes.
        NC_EMULTIDEFINE_OMODE      - create/open mode
        NC_EMULTIDEFINE_DIM_NUM    - number of dimensions
        NC_EMULTIDEFINE_DIM_SIZE   - size of dimension
        NC_EMULTIDEFINE_DIM_NAME   - dimension names
        NC_EMULTIDEFINE_VAR_NUM    - number of variables
        NC_EMULTIDEFINE_VAR_NAME   - variable name
        NC_EMULTIDEFINE_VAR_NDIMS  - variable's number of dimensions
        NC_EMULTIDEFINE_VAR_DIMIDS - variable's dimid
        NC_EMULTIDEFINE_VAR_TYPE   - variable's data type
        NC_EMULTIDEFINE_VAR_LEN    - variable's size
        NC_EMULTIDEFINE_NUMRECS    - number of records
        NC_EMULTIDEFINE_VAR_BEGIN  - variable file begin offset
        NC_EMULTIDEFINE_ATTR_NUM   - number of attributes
        NC_EMULTIDEFINE_ATTR_SIZE  - memory space used by attribute
        NC_EMULTIDEFINE_ATTR_NAME  - attribute name
        NC_EMULTIDEFINE_ATTR_TYPE  - attribute type
        NC_EMULTIDEFINE_ATTR_LEN   - attribute length
        NC_EMULTIDEFINE_ATTR_VAL   - attribute value

  o New run-time environment variables
    * PNETCDF_SAFE_MODE environment variable can be used to enable/disable the
      internal checking for data/argument consistency across all processes (by
      calling collective MPI_Allreduce). Set it to 1 to enable the checking.
      Default is 0, i.e. disabled.
    * PNETCDF_HINTS environment variable can be used to pass the I/O hints to
      PnetCDF library. Hints include both PnetCDF and MPI-IO hints.
      PNETCDF_HINTS is a string of hints separated by ";" and each hint is in
      the form of hint=value. E.g.
          romio_ds_write=disable;nc_header_align_size=1048576
      If this environment variable is set, it overrides any values that
      were set by using calls to MPI_Info_set in the application code.

  o New example programs
    * example programs are now categorized into C, F77, and F90 directories
    * nonblocking_write.f and nonblocking_write.f90 are the Fortran version of
      nonblocking_write.c
    * put_varn_float.c for using the new APIs ncmpi_put_varn_float_all()
      put_varn_real.f and put_varn_real.f90 are the Fortran versions
    * put_varn_int.c, put_varn_int.f, and put_varn_int.f90, for using the new
      APIs ncmpi_put_varn_int_all() and nfmpi_put_varn_int_all()
    * hints.c, hints.f, and hints.f90 for using PnetCDF hints
    * flexible_api.c, flexible_api.f, and flexible_api.f90, for using blocking
      and nonblocking flexible APIs
    * mput.c for using ncmpi_mput_vara_all() to write a series of arbitrary
      start[] and count[]
    * block_cyclic.c, block_cyclic.f, and block_cyclic.f90 are for a
      *-(block-cyclic) 2D partitioning pattern
    * column_wise.c, for a *-cyclic 2D partitioning pattern
    * put_vara.c, put_vara.f, and put_var.f90 for using for
      nfmpi_put_vara_int_all()

  o New programs for I/O benchmarks
    * benchmarks/C/aggregation.c -- evaluate PnetCDF's performance on I/O
      aggregation across multiple requests with different data access patterns.
    * benchmarks/C/write_block_read_column.c -- writes variables and reads back
      using different data partitioning patterns
    * benchmarks/FLASH-IO -- I/O kernel of FLASH, a reacting hydrodynamics code
      developed at University of Chicago. This benchmark can be built
      independently from PnetCDF release.

  o New test program
    * test/F90 contains test programs adopted from netcdf-fortran-4.2
    * test/nf90_test contains test programs adopted from test/nf_test
    * testcases/alignment_test.c -- test for header and fixed variable file
      offset alignments when entering redef mode
    * testcases/nonblocking.c -- test nonblocking APIs ncmpi_iput_vara_int()
      and ncmpi_iget_vara_int()
    * testcases/flexible.c -- test flexible API ncmpi_get_vara_int_all() using
      an MPI derived data type created by MPI_Type_create_hindex()
    * test/header/header_consistency.c -- test header inconsistency and see if
      inconsistent metadata is overwritten by root's

  o New semantics for attribute APIs in data mode
    The following APIs can modify file header while being called in data mode.
    Note that these APIs can only modify existing attributes, rename variables
    or dimensions, given that the new attributes and names do not take more
    file space than the old ones. Otherwise, these APIs are prohibited in data
    mode.
      ncmpi_rename_dim(),
      ncmpi_rename_var(),
      ncmpi_copy_att(),
      ncmpi_rename_att(), and
      ncmpi_put_att_<type>()
    Starting from this release of PnetCDF, these APIs must be called
    collectively when in data mode. This new requirement is to ensure the
    file header cached in memory is consistent across all processes.

  o New synchronization for number of records
    In collective data mode, the number of records cached in memory is
    always synchronized across all processes. In independent mode, the value
    can be inconsistent, unless ncmpi_sync_numrecs() (a collective API) is
    called explicitly. Otherwise, the synchronization will have to wait until
    the call to ncmpi_end_indep_data().

    Flushing the number of records to file when it is changed used to be
    delayed until ncmpi_close() or ncmpi_sync() is called. If a strong file
    consistency is desired, users must enable NC_SHARE mode when opening the
    file. However, note that in this mode any header change will cause file
    I/O to flush  the dirty header data (not just number of records). For
    programs that do not change header frequently, enabling NC_SHARE should
    have no significant performance impact.

    This flushing behavior has been changed in 1.4.0 to the following. At the
    end of collective calls, if the number of records increases, the changed
    value will be written to the file, no matter if NC_SHARE mode is used or
    not. This change is to ensure the number of records is up-to-dated in
    file, in case the application program does not close file (due to crash
    or programming error). Note this change applies to collective APIs only.

  o New optimization: I/O request aggregation
    The original design of nonblocking I/O is to concatenate the fileviews of
    individual nonblocking requests and serve them with a single MPI-IO call,
    if possible. However, because MPI-IO requires the file displacements of
    the flattened fileview be in an monotonically nondecreasing order, the
    original approach (in v1.3.1 and prior) divides the nonblocking requests
    into groups such that each group abides by this MPI-IO fileview
    requirement. Each group is then carried out by a separate MPI-IO call.
    Performance can be poor if there are multiple groups and each group's
    aggregate access region is non-contiguous in the file.

    This revision fixes this problem by 1) sorting the starting offset of all
    nonblocking requests into a non-decreasing order; 2) dividing the requests
    into groups (two types of groups are identified: interleaving and
    non-interleaving); 3) for each non-interleaving group, concatenating
    fileviews of all requests in the group; 4) for each interleaving group,
    flattening fileviews of all requests in the group, merging the
    offset-length pairs, and concatenating them into a new integrated fileview;
    5) concatenating the fileviews of all groups into a single one; 6) the
    final combined fileview is used by a single MPI-IO call to carry out the
    requests.  Performance is expected to be improved as the number of MPI-IO
    calls is reduced to one.

    However, be warned about the additional memory requirement. The additional
    memory needed for flattening the fileviews might be more than the I/O data
    itself. For example, a request to accessing a single column of a 2D integer
    array will result in offset-length pairs, each representing only a 4-byte
    integer where the C struct in PnetCDF for storing an offset-length pair
    takes 3*sizeof(MPI_Offset)=24 bytes (offset, length, and I/O buffer
    pointer).

  o Other updates:
    * configure.in and Makefile.in have been revised to detect MPI compilers
      and other compile options automatically.
    * A new configure option "--disable-file-sync" to disable calling file
      sync. This is to be used when the underlying file system provides data
      consistency control.
    * add build recipe for IBM BGQ (e.g. Vesta/Mira/Cetus @ANL) in README.IBM
    * add build recipe for CRAY-XE6, Hopper @NERSC in README.CRAY
    * add build recipe for CRAY-XC30, Edison @NERSC in README.CRAY
    * add build recipe for Endeavour @ NASA in README.SGI
    * add declaration of flexible APIs for Fortran90
    * "make testing" now hides most of the stdout. Use "make verbose_testing"
      for verbose output.
    * ncmpidump: add the command-line option "-k" to report the format of
      netCDF file.
    * ncvalid is renamed to ncmpivalid, a tool to validate the structure of
      netCDF files for conforming with CDF formats.
    * Fortran type NFMPI_OFFSET is removed. It was merely a shortcut to
      integer(KIND=MPI_OFFSET_KIND). Some Fortran 77 compiler does not
      recognize keyword KIND. In that case, please set MPIF77 to the MPI
      Fortran 90 compiler, e.g. ./configure MPIF77=mpif90
    * configure now automatically checks Fortran module compile flags
    * Support additional Fortran netCDF data types: nf_ubyte, nf_ushort,
      nf_uint, nf_uint64, nf90_ubyte, nf90_ushort, nf90_uint, nf90_uint64.
    * Error codes and messages conform with netCDF's

  o Bug fixes
    * Argument unlimdimid of nfmpi_inq() returns -1 when no unlimited length
      dimension has been defined (to conform nf_inq()).
    * Argument varid of nfmpi_inq_varoffset() is fixed to be the C's varid
      plus one.
    * For collective APIs, many places have been changed to prevent program
      from hanging if a subset of the processes got errors. The fix will allow
      all processes participating the MPI collective calls in the PnetCDF, even
      if errors are detected on a subset of processes.
    * set the nonblocking request ID to NULL when the request length is zero
    * report error when bogus request IDs are passed in ncmpi_wait
    * when entering redef with different alignment hints, fixed-size
      variables' file starting offsets will only be changed when it is bigger
      than the old ones
    * Fix some Fortran API intent in/out argument declarations
    * ncmpi_def_var is fixed to detect if CDF-5 data types is used on CDF-1 or
      CDF-2 files. Error code NC_ESTRICTNC3 will return.



-------------------------------------
version  1.3.1 (September 24, 2012)
-------------------------------------

    This release is primarily a bug-fix release, tidying up a few issues and
    incorporating some early feedback on our "buffered put" interface (see
    http://trac.mcs.anl.gov/projects/parallel-netcdf/wiki/BufferedInterface for
    more information)

    - add a new API ncmpi_inq_buffer_usage/nfmpi_inq_buffer_usage for inquiring
      the current usage of the internal buffer allocated by the "buffered"-put
      APIs.
    - bug fix to make bput APIs properly return error code NC_EINSUFFBUF.
    - bug fixes for ncmpidump to avoid residue contents from a previous read
      when it read beyond EOF.
    - bug fixes in the tutorial example codes.
    - add more in-line comments for the tutorial example codes.
    - add the error string for error code NC_ENOENT.

-------------------------------------
version  1.3.0 (June 26, 2012)
-------------------------------------

    - Bug fixes in new ncmpidiff tool.
    - Small optimizations to reduce communication in library.
    - Improved documentation, including more test programs and a QuickTutorial.
    - Bug fixes in our Fortran90 support.
    - Better compatibility with NetCDF-4: no need for a modified pnetcdf.h from
      Unidata when building NetCDF-4 with PnetCDF support.
    - PnetCDF now duplicates the MPI communicator internally, which fixed at
      least one odd behavior seen in a PnetCDF-using application.
    - Improvements to PnetCDF header and variable alignment (see wiki page
      VariableAlignment).
    - Add a checking for file create mode consistency across all processes and
      an error code for it, NC_ECMODE.
    - Bug fix for updating the number of record variables in nonblocking APIs.
    - Bug fix for variable starting file offsets when defining multiple large
      variables (> 232 elements) in one CDF-5 file.
    - Add a new configure option that takes environment variable $RM to replace
      the default "rm" command in all Makefiles.
    - Bug fix for nonblocking varm APIs.
    - Support for CDF-2's special2 characters in names (variables, attributes,
      dimensions, etc.)
    - Release of the official CDF-5 file format specification (see wiki page
      CD-5).
    - Support for CDF-5 data types: NC_UBYTE, NC_USHORT, NC_UINT, NC_INT64,
      NC_UINT64, and NF_INT64.
    - New C APIs: ncmpi_put_vara_ushort, ncmpi_put_vara_uint,
      ncmpi_put_vara_longlong, and ncmpi_put_vara_ulonglong. Similarly for
      var1, var, vars and varm APIs. Also for get and nonblocking APIs.
    - New Fortran APIs: nfmpi_put_vars_int8 and similarly for var1, var, vars,
      varm, get, and nonblocking APIs.
    - Add a new error code, NC_ENOTSUPPORT, for not-yet-supported features.
    - A new set of buffered put APIs (eg. ncmpi_bput_vara_float) is added (see
      BufferedInterface). They make a copy of the user's buffer internally, so
      the user's buffer can be reused when the call returns. Their usage are
      similar to the iput APIs.
    - Add new error codes for buffered put APIs: NC_ENULLBUF,
      NC_EPREVATTACHBUF, NC_ENULLABUF, NC_EPENDINGBPUT, and NC_EINSUFFBUF.
    - Add new test and example programs for the buffered put APIs.
    - The error string returned by ncmpi_strerror() for an undefined error code
      is updated from "Unknown Error" to "Unknown Error: Unrecognized PnetCDF
      error code ID" to be more descriptive.
    - Remove the use of POSIX error codes: EINVAL, ENOERR, ERANGE, and ENOMEM.
      They are replaced by NC error codes now: NC_EINVAL, NC_NOERR, NC_ERANGE,
      and NC_ENOMEM.
    - A better translation of MPI-IO error classes to netCDF/PnetCDF error
      codes. For example, MPI-IO error class MPI_ERR_NO_SUCH_FILE is translated
      to NC_ENOENT.

    - Compatibility note

    - In testing with version 11.7 of the Portland Group compiler, some of the
      Fortran test programs fail to compile if built with pgf77. The tests work
      if built with the Fortran90 compiler.

-------------------------------------
version  1.2.0 (August 19, 2010)
-------------------------------------

    - Several bugs fixed, memory leaks reduced,

    - Internal housekeeping

    - Reduce the places where PnetCDF will call MPI_FILE_SYNC

    - A new 'ncmpidiff' utility to compare two datasets in parallel

    - Addressed a build failure when compiling with some versions of MVAPICH2.
      See
      http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2010-July/001045.html
      for an example of the build failure.

    - If you build this release and still get "undefined reference to 'yyin'",
      then you have some stale object files. "make clean" and re-build.

    - Additional error checking for collective routines

    - We have also made API changes to our non-blocking interface. Our
      non-blocking semantics have been relaxed somewhat, and should be more
      usable in more situations.


-------------------------------------
version  1.1.0 (November 2, 2009)
-------------------------------------

    - A new file format, CDF-5, is introduced. This format allows defining
      large array variables with more than 2^32 elements.

    - A new optimization in parallel-netcdf with this release. If the hint
      "striping_unit" is set, then PnetCDF will align the start of non-record
      variables to a multiple of that value.

    - A new set of APIs for reading/writing multiple variables. The existing
      asynchronous APIs have also been improved to enable combination of
      multiple variable access into fewer I/O requests.

    - There is now a simple pnetcdf.F90 module for F90 codes. Please consider
      this as an "early feedback" version and not a hard guarantee of a fixed
      F90 API (though we don't expect it to change drastically if at all).
      This module is generated from the F77 'pnetcdf.inc': thanks to
      Annette Koontz for the idea.

    - The 'ncmpigen' utility can now create "big variable" (CDF-5) files, in
      addition to the older "big file" (CDF-2) files.

-------------------------------------
version  1.0.0 (July 27, 2005)
-------------------------------------

    - The Parallel-NetCDF developers are quite happy to announce our 1.0.0
      release.  This release reflects the culmination of several years of
      work and lots of community feedback.  

      A brief list of the major changes since 0.9.4:

    - Has both the high level and flexible data mode interfaces 
    - Improved support for many more platforms
    - Support for all serial NetCDF access patterns (var, vara, vars,
      varm) and types (text, char, uchar, schar, short, int, float, long,
      double)
    - Synced up with netcdf-3.6.0
    - The usual array of bug fixes and improvements.

