Table of Contents

Node:Top, Next:, Up:(dir)

mpatrol

images/mpatrol.jpg

This document describes mpatrol, a library for controlling and tracing dynamic memory allocations.

This is edition 2.13 of the mpatrol manual for version 1.4.8, 8th January, 2002.

Node:Foreword, Next:, Previous:Top, Up:Top

Foreword

I first started writing this library a few years ago when the company I work for sent me out to a customer who had reported a memory leak, which he expected was coming from the code generated by our C++ compiler. A few years on and the library has changed dramatically from its first beginnings, but I thought I'd release it publicly in case anyone else found it useful.

When writing the library, I placed more emphasis on the quantity and quality of information about allocated memory rather than the speed and efficiency of allocating the actual memory. This means that the library will use dramatically more memory than normal dynamic memory allocation libraries and can slow down to a crawl depending on which options you use. However, the end results are likely to be accurate and reliable, and in most cases the library will run quite happily at a sane speed.

The mpatrol library is by no means the only library of its kind. Solaris has no less that 6 different malloc libraries, and there are plenty available as freeware or as commercial products. Try to keep in mind that mpatrol comes with absolutely no warranty and so if it doesn't work for you and you need a fast solution, try some of the other libraries or products available. I have listed some of the most popular at the end of this manual (see Related software).

This manual is arranged so that complete reference material on the mpatrol library can be found in the appendices, while introductory and background material can be found in the preceding chapters and sections. For readers who wish to delve right in and use the library, the Installation (see Installation) and Examples (see Examples) chapters should be enough to get started in combination with the quick reference card. Otherwise, this manual should be read from beginning to end in order to get the most out of the software it describes. Note that all of the output shown from the examples was produced on 32-bit environments, although mpatrol can be built to support 64-bit environments as well.

Due to their very nature, problems with dynamic memory allocations are notoriously difficult to reproduce and debug, and this is likely to be the case if you find a bug in the mpatrol library as it might be extremely hard to reproduce on another system. Details on how to report bugs are given elsewhere in this document (see Notes), but it would be very useful if you could try to provide as much information as possible when reporting a problem, and that includes having a look in the library source code to see if it's obvious what is wrong. However, please try to read the frequently asked questions (see Frequently asked questions) first in case your question or problem is covered there since they are usually updated every time I receive a question about mpatrol.

The latest version of the mpatrol library and this manual can always be found at http://www.cbmamiga.demon.co.uk/mpatrol/, and any correspondence relating to mpatrol (bug reports, enhancement requests, compliments, etc.) should be sent to mpatrol@cbmamiga.demon.co.uk. I'd be very interested in hearing any success stories with using mpatrol to debug programs, since I get very little feedback apart from the occasional bug report. The mpatrol library is also registered at FreshMeat (http://freshmeat.net/projects/mpatrol/) and SourceForge (http://sourceforge.net/projects/mpatrol/) and several other software sites so you can receive notification of updates there as well. I normally only check my e-mail about once or twice a week, so don't expect an immediate response. I can also be reached at graeme.roy@analog.com but that is my work e-mail address. There is also a discussion group at http://groups.yahoo.com/group/mpatrol/ where you can post mpatrol-related questions but you must first subscribe to the group before you can send mail to it.

Finally, I'd like to thank Stephan Springl (springl@bfw-online.de) for his help on reading debugging information from object files via the GNU BFD library, and Adam Zell (zell@best.com) for helping with patching the dynamic linker support functions for loading shared libraries. Both Alexander Barton (abarton@innotrac.com) and Dave Gibson (david.gibson@analog.com) helped to make mpatrol thread-safe. Roger Keane (rgr@bcs-inc.com) provided the perl code in the mpsym command and also the idea for the MP_USE_ATEXIT feature macro. Steve McIntyre (smcintyre@allstor-sw.co.uk) helped by diagnosing the fork() problem and provided example code on how to fix it. Peter Zijlstra (peter@xlnt-software.com) contributed code to enhance stack traces for errors.

Boris Makushkin (oberon@antibiotic.ru) requested, helped with, and provided initial testing for the FreeBSD port, and Ivan Finch (i.finch@rl.ac.uk), Gerrit Bruchhaeuser (gbruchhaeuser@orga.com) and Andreas Schallenberg (andreas.schallenberg@informatik.uni-oldenburg.de) did the same for the Tru64 and SuSE ports. Both Aleksandar Donev (donev@pa.msu.edu) and Van Snyder (vsnyder@math.jpl.nasa.gov) provided suggestions and code for using mpatrol with FORTRAN. Michael Anthony (m@xyzfind.com) wrote the profdiff tool and Jerome Marant (jerome@debian.org) did the Debian GNU/Linux port.

In addition, after spending well over 2000 hours designing and programming mpatrol, my sanity would not have been preserved in the state it is in today were it not for the music of The Chemical Brothers, The Manic Street Preachers, Orbital and The Prodigy. You can now argue how sane I am depending on your musical tastes!

Oh, and always remember to do final release builds without the mpatrol library as the library is much slower than normal malloc implementations and uses much more memory.

Happy debugging!

Graeme Roy, 11th October, 1999.

Edinburgh, Scotland.

Node:Overview, Next:, Previous:Foreword, Up:Top

1 Overview

The mpatrol library is yet another link library that attempts to diagnose run-time errors that are caused by the wrong use of dynamically allocated memory. If you don't know what the malloc() function or operator new[] do then this library is probably not for you. You have to have a certain amount of programming expertise and a knowledge of how to run a command line compiler and linker before you should attempt to use this.

Along with providing a comprehensive and configurable log of all dynamic memory operations that occurred during the lifetime of a program, the mpatrol library performs extensive checking to detect any misuse of dynamically allocated memory. All of this functionality can be integrated into existing code through the inclusion of a single header file at compile-time. On UNIX and Windows platforms (and AmigaOS when using gcc) this may not even be necessary as the mpatrol library can be linked with existing object files at link-time or, on some platforms, even dynamically linked with existing programs at run-time.

All logging and tracing output from the mpatrol library is sent to a separate log file in order to keep its diagnostics separate from any that the program being tested might generate. A wide variety of library settings can also be changed at run-time via an environment variable, thus removing the need to recompile or relink in order to change the library's behaviour.

A file containing a summary of the memory allocation profiling statistics for a particular program can be produced by the mpatrol library. This file can then be read by a profiling tool which will display a set of tables based upon the accumulated data. The profiling information includes summaries of all of the memory allocations listed by size and the function that allocated them and a list of memory leaks with the call stack of the allocating function. It also includes a graph of all memory allocations listed in tabular form, and an optional graph specification file for later processing by the dot graph visualisation package.

A file containing a concise encoded trace of all memory allocations, reallocations and deallocations made by a program can also be produced by the mpatrol library. This file can then be read by a tracing tool which will decode the trace and display the events in tabular or graphical form, and also display any relevant statistics that could be calculated.

The mpatrol library has been designed with the intention of replacing calls to existing C and C++ memory allocation functions as seamlessly as possible, but in many cases that may not be possible and slight code modifications may be required. However, a preprocessor macro containing the version of the mpatrol library is provided for the purposes of conditional compilation so that release builds and debug builds can be easily automated.

Node:Features, Next:, Previous:Overview, Up:Top

2 Features

An overall list of features contained in the mpatrol library is given below. This is not intended to be exhaustive since the best way to see what the library does is to read the documentation and try it out.

Node:Installation, Next:, Previous:Features, Up:Top

3 Installation

The mpatrol library was initially developed on an Amiga 4000/040 running AmigaOS 3.1. I then installed Red Hat Linux 5.1 on my Amiga and added support for Linux/m68k. I've now just recently purchased a Dell Inspiron 7500 Notebook PC and put my Amiga in retirement, so development will now continue on Red Hat Linux 6.2 and above on the Intel platform. I've tried my best to make it as easy as possible to build and install mpatrol on any system, but it isn't likely to run smoothly for everybody. However, there shouldn't be any major problems if you perform the following steps.

Note that if you want to check the integrity of the files that came with the mpatrol distribution you can use the CHECKSUMS file in the mpatrol base directory. You must have the md5sum command installed on your system in order to make use of this file.

If you wish to use GNU autoconf, automake and libtool to build and install mpatrol you may do so by entering the pkg/auto directory and typing ./setup. This will construct the directory structure that is required by these tools and will also create a configure script. Please see the INSTALL file in that directory for information on how to proceed. Note that you can clean up the pkg/auto directory by typing make distclean (if the configure script has already been run) followed by ./cleanup.

For a manual installation, perform the following steps.

  1. Go into the build directory and then into the appropriate subdirectory for your system.
  2. Edit the Makefile in that directory and check that it is using the appropriate compiler and build tools. The CC macro specifies the compiler1, the AR macro specifies the tool used to build the archive library and the LD macro specifies the tool to build the shared library. The CFLAGS macro specifies compiler options that are always to be used, the OFLAGS macro specifies optimisation options for the compiler, the SFLAGS macro specifies options to be passed to the compiler when building a shared library and the TFLAGS macro specifies options to be passed to the compiler when building a thread-safe library. You may also need to change the library names and library build commands on different systems.

    Note that the generic UNIX Makefile contains a macro called GUISUP which is set to false by default. If it is set to true then the mptrace command will be built with GUI support enabled. However, your system must contain the correct header files and libraries in order to support this.

  3. Use the make command (or equivalent) to build the mpatrol library in archive form. The all target builds all possible combinations of the mpatrol library for your system. The clean target removes all relevant object files from the current directory, while the clobber target also removes all libraries that have been built from the current directory. On some UNIX platforms, the lint target will build a lint library for the mpatrol library.
  4. If the mpatrol library is to be built with support for Parasoft Inuse then the MP_INUSE_SUPPORT preprocessor macro must be defined in the CFLAGS portion of the Makefile before building. This will ensure that Inuse will be notified of every memory allocation, reallocation and deallocation, but the Insure++ runtime library will also have to be linked in with any program that uses mpatrol.
  5. Copy all of the libraries that have been built into your local library directory. If there were symbolic links created in the build directory then these should be recreated in the local library directory rather than simply copying them. You may need to run a command such as ldconfig in order for the system to recognise the newly-installed libraries, and you may also need to add the filename of the directory containing the newly-installed libraries to an environment variable such as LD_LIBRARY_PATH if you installed the libraries in a non-standard location.
  6. Copy the mpatrol, mprof, mptrace and mleak programs that have been built into your local bin directory. You may also wish to copy the mpsym, mpedit and hexwords commands to your local bin directory as well if your system supports Bourne shell scripts.
  7. Go up two directory levels into the src directory and copy the mpatrol.h, mpalloc.h and mpdebug.h header files into your local include directory.
  8. Go up one directory level into the tools directory and copy all of the header files into the mpatrol subdirectory (which you'll need to create) in your local include directory.
  9. On UNIX platforms, go up one directory level into the man directory and copy the man1 and man3 subdirectories to your local man directory. Unfortunately, the location for manual pages varies from system to system so you may or may not also be able to copy the cat1 and cat3 subdirectories as well. The man* subdirectories contain the unformatted manual pages while the cat* subdirectories contain the formatted manual pages.
  10. Go up one directory level into the doc directory and examine the files located there. The mpatrol.texi file contains the TeXinfo source for this manual and can be translated into a wide variety of documentation formats. The refcard.tex file contains the LaTeX source for the quick reference card and can be translated into formats suitable for printing onto a few pages. There should already be translated files in the doc directory, but if not you will have to generate them yourself using the Makefile provided. You can then install or print these documents.

    The mpatrol library source code can also be formatted for a printed manual for later perusal. The source target in the Makefile within the doc directory can be used to build the source code documentation in DVI, postscript and PDF formats, but be prepared for a large number of pages!

    If you are not installing on a system that supports UNIX manual pages then you should also check in the man directory to ensure that there are alternative formats for the mpatrol manual pages that you can install. If not, you will have to generate them yourself using the Makefile provided.

Alternatively, the pkg directory contains files that can be used to automatically generate a package in a specific format suitable for installation on a system. Four package formats (PKG, SD/UX, RPM and Debian) and three archive formats are currently supported (generic tape archive, LhA and ZIP).

The first package format is generally used on UNIX SVR4 systems, while the second is used on HP/UX systems. The RPM and Debian package formats were introduced by Red Hat and Debian respectively for use in their Linux distributions.

The generic tape archive can be used as a distribution for UNIX systems where no package format is supported, but it does not contain information on how to install the files on the system once they have been extracted from the distribution. The LhA and ZIP formats are also roughly the same, but the LhA format is intended for Amiga systems and is used for Aminet distributions, while the ZIP format is intended for Windows systems and is used for WinSite distributions.

You should really know what you are doing before you attempt to build a package, and you should also be aware that some of the package files may need to be modified before you begin.

In addition, a Linux Software Map index file exists in the pkg/lsm directory.

Note that the extra directory that comes with the mpatrol distribution contains several prototype configuration files for certain third-party programs. These files should be examined so that you can decide whether to integrate their contents into your existing configuration files. The purpose of each file is described in the relevant sections of this manual.

Node:Integration, Next:, Previous:Installation, Up:Top

4 Integration

This section describes how to go about adding or removing the mpatrol library from your code. There are several levels for each category so it is worth reading about each before proceeding.

Node:Adding mpatrol, Next:, Up:Integration

4.1 Adding mpatrol

The following steps should allow you to easily integrate the mpatrol library into an existing application, although some of them may not be available to do on many platforms. They are listed in the order of number of changes required to modify existing code -- the last step will require a complete recompilation of all your code.

  1. This step is currently only available on DYNIX/ptx, FreeBSD, IRIX, Linux, NetBSD, OpenBSD, Solaris and Tru64 platforms and on DG/UX 4.20MU072 or later platforms with the LD_PRELOAD feature.

    If your program or application has been dynamically linked with the system C library (libc.so) or an alternative malloc shared library then you can use the --dynamic option to the mpatrol command to override the default definitions of malloc(), etc. at run-time without having to relink your program. If your program is multithreaded then you must also add the --threads option to pick up the multithreaded shared libraries instead.

    For example, if your program's executable file is called testprog and it accepts an option specifying an input file, you can force the system's dynamic linker to use mpatrol's versions of malloc(), etc. instead of the default versions by typing: 

    mpatrol --dynamic ./testprog -i file

    The resulting log file should be called mpatrol.<procid>.log by default (where procid is the current process id), but if no such file exists after running the mpatrol command then it will not be possible to force the run-time linking of mpatrol functions to your program and you will have to proceed to the next step. Note that the mpatrol command overrides any previous setting of the MPATROL_OPTIONS environment variable.

  2. This step is currently only available on UNIX and Windows platforms (and AmigaOS when using gcc).

    You should be able to link in the mpatrol library when linking your program without having to recompile any of your object files or libraries, but this will only be worthwhile on systems where stack tracebacks are supported, otherwise you should proceed to the next step since there will not be enough information for you to tell where the calls to dynamic memory allocation functions took place.

    Information on how to link the mpatrol library to an application is given at the start of the examples (see Examples), but you should note that if your program does not directly call any of the functions in the mpatrol library then it will not be linked in and you will not see a log file being generated when you run it. You can force the linking of the mpatrol library by causing malloc() to be undefined on the link line, usually through the use of the -u linker option.

    If your program is multithreaded then you must use the thread-safe version of the mpatrol library and possibly also link in the system threads library as well. Not doing this will usually result in your program failing somewhere in the mpatrol library code.

  3. All of the following steps will require you to recompile some or all of your code so that your code calls dynamic memory allocation functions from the mpatrol library rather than the system C library.

    This first step is only available when using gcc.

    You can make use of the gcc option -fcheck-memory-usage which instructs the compiler to place calls to error-checking functions before each access to memory. This can result in a dramatic slowdown of your code so you may wish to limit the use of this option to a few source files, but it does provide a very thorough method of ensuring that you do not access memory beyond the bounds of a memory allocation or attempt to access free memory. However, be aware that the checks are only placed in the bodies of functions that have been compiled with this option and are missing from all functions that have not. You must link in the mpatrol library when using this option, otherwise you will get linker errors.

    The -fcheck-memory-usage option was added to gcc to support GNU Checker, which can be considered to be the run-time system for this option. GNU Checker also includes the ability to detect reads from uninitialised memory, something that mpatrol does not currently support, and deals with stack objects as well. GNU Checker cannot be used in conjunction with mpatrol.

  4. For this step, if you have a rough idea of where the function calls lie that you would like to trace or test, you need only recompile the relevant source files. You should modify these source files to include the mpatrol.h header file before any calls to dynamic memory allocation or memory operation functions.

    However, you should take particular care to ensure that all calls to memory allocation functions in the mpatrol library will be matched by calls to memory reallocation or deallocation functions in the mpatrol library, since if they are unmatched then the log file will either fill up with errors complaining about trying to free unknown allocations, or warnings about unfreed memory allocations at the end of execution.

  5. This step requires you to recompile all of your source files to include the mpatrol.h header file. Obviously, this will take the longest amount of time to integrate, but need not require you to change any source files if the compiler you are using has a command line option to include a specific header file before any source files.

    For example, gcc comes with a -include option which has this feature, so if you had to recompile a source file called test.c then the following command would allow you to include mpatrol.h without having to modify the source file: 

    gcc -include /usr/local/include/mpatrol.h -c test.c

In all cases, it will be desirable to compile your source files with compiler-generated debugging information since that may be able to be used by the USEDEBUG option or the mpsym command. In addition, more symbolic information will be available if the executable files have not had their symbol tables stripped from them, although mpatrol can also fall back to using the dynamic symbol table from dynamically linked executable files.

Note that an automake macro is now provided to allow you to integrate mpatrol into a new or existing project that uses the GNU autoconf and automake tools. It is located in extra/mpatrol.m4, which should be copied to the directory containing all of the local autoconf and automake macros on your system, usually /usr/local/share/aclocal. The automake macro it defines is called AM_WITH_MPATROL, which should be added to the libraries section in the configure.in file for your project. It takes one optional parameter specifying whether mpatrol should be included in the project (yes) or not (no). This can also be specified as threads if you wish to use the threadsafe version of the mpatrol library. You can override the value of the optional parameter with the --with-mpatrol option to the resulting configure shell script.

If you are using the AM_WITH_MPATROL automake macro then you may wish to use the mpdebug.h header file instead of mpatrol.h. This ensures that the MP_MALLOC() family of functions are always defined, even if libmpatrol or libmpalloc are unavailable. It makes use of the HAVE_MPATROL and HAVE_MPALLOC preprocessor macros that are controlled by the automake macro, but in other respects behaves in exactly the same way as mpatrol.h.

Node:Removing mpatrol, Previous:Adding mpatrol, Up:Integration

4.2 Removing mpatrol

Once you have ironed out all of the problems in your code with the help of the mpatrol library, there will come a time where you wish to build your program without any of its debugging features, either to improve the speed that it runs at, or perhaps even for a release. Choose one of the following steps to help you remove the mpatrol library from your program (you only need to perform them if you linked your program with the mpatrol library).

  1. The quickest way to remove the mpatrol library from your application is to link with libmpalloc instead of libmpatrol. This contains replacements for all of the mpatrol library functions, either implementing memory allocation or memory operation functions with the system C library, or doing nothing in the functions which perform debugging, profiling or tracing. This method is a very quick way to remove the mpatrol library but will not result in very efficient code.
  2. The next option is to recompile all of the source files which include the mpatrol.h header file, but this time define the NDEBUG preprocessor macro. This automatically disables the redefinition of malloc(), etc. and prevents calls being made to any mpatrol library functions. Obviously, this option is the most time-consuming of the two, but will result in the complete removal of all references to the mpatrol library.
  3. The final option is to guard all of the mpatrol-specific code in your program with a preprocessor macro, possibly called HAVE_MPATROL, and then recompiling all of your source code with this macro undefined. This is the best option but relies on you having originally made these changes when you first started integrating the mpatrol library into your program.

Note that if you used the AM_WITH_MPATROL automake macro as detailed in the previous section to build your application then you should perform a clean recompilation using the --without-mpatrol option to the configure shell script in order to completely remove the mpatrol library.

Note also that if you used the -fcheck-memory-usage option of the GNU compiler to check all memory accesses then you must recompile without that option in order for your program to run at a reasonable speed.

Node:Memory allocations, Next:, Previous:Integration, Up:Top

5 Memory allocations

In the C and C++ programming languages there are generally three different types of memory allocation that can be used to hold the contents of variables. Other programming languages such as Pascal, BASIC and FORTRAN also support some of these types of allocation, although their implementations may be slightly different.

Node:Static memory allocations, Next:, Up:Memory allocations

5.1 Static memory allocations

The first type of memory allocation is known as a static memory allocation, which corresponds to file scope variables and local static variables. The addresses and sizes of these allocations are fixed at the time of compilation3 and so they can be placed in a fixed-sized data area which then corresponds to a section within the final linked executable file. Such memory allocations are called static because they do not vary in location or size during the lifetime of the program.

There can be many types of data sections within an executable file; the three most common are normal data, BSS data and read-only data. BSS data contains variables and arrays which are to be initialised to zero at run-time and so is treated as a special case, since the actual contents of the section need not be stored in the executable file. Read-only data consists of constant variables and arrays whose contents are guaranteed not to change when a program is being run. For example, on a typical SVR4 UNIX system the following variable definitions would result in them being placed in the following sections: 

int a;           /* BSS data */
int b = 1;       /* normal data */
const int c = 2; /* read-only data */

In C the first example would be considered a tentative declaration, and if there was no subsequent definition of that variable in the current translation unit then it would become a common variable in the resulting object file. When the object file gets linked with other object files, any common variables with the same name become one variable, or take their definition from a non-tentative definition of that variable. In the former case, the variable is placed in the BSS section. Note that C++ has no support for tentative declarations.

As all static memory allocations have sizes and address offsets that are known at compile-time and are explicitly initialised, there is very little that can go wrong with them. Data can be read or written past the end of such variables, but that is a common problem with all memory allocations and is generally easy to locate in that case. On systems that separate read-only data from normal data, writing to a read-only variable can be quickly diagnosed at run-time.

Node:Stack memory allocations, Next:, Previous:Static memory allocations, Up:Memory allocations

5.2 Stack memory allocations

The second type of memory allocation is known as a stack memory allocation, which corresponds to non-static local variables and call-by-value parameter variables. The sizes of these allocations are fixed at the time of compilation but their addresses will vary depending on when the function which defines them is called. Their contents are not immediately initialised, and must be explicitly initialised by the programmer upon entry to the function or when they become visible in scope.

Such memory allocations are placed in a system memory area called the stack, which is allocated per process4 and generally grows down in memory. When a function is called, the state of the calling function must be preserved so that when the called function returns, the calling function can resume execution. That state is stored on the stack, including all local variables and parameters. The compiler generates code to increase the size of the stack upon entry to a function, and decrease the size of the stack upon exit from a function, as well as saving and restoring the values of registers.

There are a few common problems using stack memory allocations, and most generally involve uninitialised variables, which a good compiler can usually diagnose at compile-time. Some compilers also have options to initialise all local variables with a bit pattern so that uninitialised stack variables will cause program faults at run-time. As with static memory allocations, there can be problems with reading or writing past the end of stack variables, but as their sizes are fixed these can usually easily be located.

Node:Dynamic memory allocations, Previous:Stack memory allocations, Up:Memory allocations

5.3 Dynamic memory allocations

The last type of memory allocation is known as a dynamic memory allocation, which corresponds to memory allocated via malloc() or operator new[]. The sizes, addresses and contents of such memory vary at run-time and so can cause a lot of problems when trying to diagnose a fault in a program. These memory allocations are called dynamic memory allocations because their location and size can vary throughout the lifetime of a program.

Such memory allocations are placed in a system memory area called the heap, which is allocated per process on some systems, but on others may be allocated directly from the system in scattered blocks. Unlike memory allocated on the stack, memory allocated on the heap is not freed when a function or scope is exited and so must be explicitly freed by the programmer. The pattern of allocations and deallocations is not guaranteed to be (and is not really expected to be) linear and so the functions that allocate memory from the heap must be able to efficiently reuse freed memory and resize existing allocated memory on request. In some programming languages there is support for a garbage collector, which attempts to automatically free memory that has had all references to it removed, but this has traditionally not been very popular for programming languages such as C and C++, and has been more widely used in functional languages like ML5.

Because dynamic memory allocations are performed at run-time rather than compile-time, they are outwith the domain of the compiler and must be implemented in a run-time package, usually as a set of functions within a linker library. Such a package manages the heap in such a way as to abstract its underlying structure from the programmer, providing a common interface to heap management on different systems. However, this malloc library must decide whether to implement a fast memory allocator, a space-conserving memory allocator, or a bit of both. It must also try to keep its own internal tables to a minimum so as to conserve memory, but this means that it has very little capability to diagnose errors if any occur.

In some compiler implementations there is a builtin function called alloca(). This is a dynamic memory allocation function that allocates memory from the stack rather than the heap, and so the memory is automatically freed when the function that called it returns. This is a non-standard feature that is not guaranteed to be present in a compiler, and indeed may not be possible to implement on some systems6. However, the mpatrol library provides a debugging version of this function (and a few other related functions) on all systems, so that they make use of the heap instead of the stack.

As can be seen from the above paragraphs, dynamic memory allocations are the types of memory allocations that can cause the most problems in a program since almost nothing about them can be used by the compiler to give the programmer useful warnings about using uninitialised variables, using freed memory, running off the end of a dynamically-allocated array, etc. It is these types of memory allocation problems that the mpatrol library loves to get its teeth into!

Node:Operating system support, Next:, Previous:Memory allocations, Up:Top

6 Operating system support

Beneath every malloc library's public interface there is the underlying operating system's memory management interface. This provides features which can be as simple as giving processes the ability to allocate a new block of memory for themselves, or it can offer advanced features such as protecting areas of memory from being read or written. Some embedded systems have no operating systems and hence no support for dynamic memory allocation, and so the malloc library must instead allocate blocks of memory from a fixed-sized array. The mpatrol library can be built to support all of the above types of system, but the more features an operating system can provide it with, the more it can do.

On operating systems such as UNIX and Windows, all dynamic memory allocation requests from a process are dealt with by using a feature called virtual memory. This means that a process cannot perform illegal requests without them being denied, which protects the other running processes and the operating system from being affected by such errors. However, on AmigaOS and Netware platforms there is no virtual memory support and so all processes effectively share the same address space as the operating system and any other running processes. This means that one process can accidentally write into the data structures of another process, usually causing the other process to fail and bring down the system. In addition, a process which allocates a lot of memory will result in there being less available memory for other running processes, and in extreme cases the operating system itself.

Node:Virtual memory, Next:, Up:Operating system support

6.1 Virtual memory

Virtual memory is an operating system feature that was originally used to provide large usable address spaces for every process on machines that had very little physical memory. It is used by an operating system to fool7 a running process into believing that it can allocate a vast amount of memory for its own purposes, although whether it is allowed to or not depends on the operating system and the permissions of the individual user.

Virtual memory works by translating a virtual address (which the process uses) into a physical address (which the operating system uses). It is generally implemented via a piece of hardware called a memory management unit, or MMU. The MMU's primary job is to translate any virtual addresses that are referred to by machine instructions into physical addresses by looking up a table which is built by the operating system. This table contains mappings to and from pages8 rather than bytes since it would otherwise be very inefficient to handle mappings between individual bytes. As a result, every virtual memory operation operates on pages, which are indivisible and are always aligned to the system page size.

Even though each process can now see a huge address space, what happens when it attempts to allocate more pages than actually physically exist, or allocate an additional page of memory when all of the physical pages are in use by it and other processes? This problem is solved by the operating system temporarily saving one or more of the least-used pages (which might not necessarily belong that that process) to a special place in the file system called a swap file, and mapping the new pages to the physical addresses where the old pages once resided. The old pages which have been swapped out are no longer currently accessible, but their location in the swap file is noted in the translation table.

However, if one of the pages that has been swapped out is accessed again, a page fault occurs at the instruction which referred to the address and the operating system catches this and reloads the page from the swap file, possibly having to swap out another page to make space for the new one. If this occurs too often then the operating system can slow down, having to constantly swap in and swap out the same pages over and over again. Such a problem is called thrashing and can only really be overcome by using less virtual memory or buying more physical memory.

It is also possible to take advantage of the virtual memory system's interaction between physical memory and the file system in program code, since mapping an existing file to memory means that the usual file I/O operations can be replaced with memory read and write operations. The operating system will work out the optimum way to read and write any buffers and it means that only one copy of the file exists in both physical memory and the file system. Note that this is how shared libraries9 on UNIX platforms are generally implemented, with each individual process that uses the shared library having it mapped to somewhere in its address space.

Another major feature of virtual memory is its ability to read protect and write protect individual pages of process memory. This means that the operating system can control access to different parts of the address space for each process, and also means that a process can read and/or write protect an area of memory when it wants to ensure that it won't ever read or write to it again. If an illegal memory access is detected then a signal will be sent to the process, which can either be caught and handled or will otherwise terminate the process. Note that as with all virtual memory operations, this ability to protect memory only applies to pages, so that it is not possible to protect individual bytes.

However, some versions of UNIX have programmable software watch points which are implemented at operating system level. These are normally used by debuggers to watch a specified area of memory that is expected to be read from or written to, but can just as easily be used to implement memory protection at byte level. Unfortunately, as this feature is implemented in software10 rather than in hardware, watch points tend to be incredibly slow, mainly as a result of the operating system having to check every instruction before it is executed. In addition, some UNIX platforms only allow a certain number of software watch points to be in use at any one time, so even if your system supports them you may not be able to use them with the mpatrol library if there are many memory allocations in use at one time.

There is also an additional problem when using watch points, which is due to misaligned reads from memory. These can occur with compiler-generated code or with optimised library routines where memory read, move or write operations have been optimised to work at word level rather than byte level. For example, the memcpy() function would normally be written to copy memory a byte at a time, but on some systems this can be improved by copying a word at a time. Unfortunately, care has to be taken when reading and writing such words as the equivalent bytes may not be aligned on word boundaries. Technically, reading additional bytes before or after a memory allocation when they share the same word is legal, but when using watch points such errors will be picked up. The mpatrol library replaces most of the memory operation functions provided by the system libraries with safer versions, although they may not be as efficient.

An operating system with virtual memory is usually going to run ever so slightly slower than an operating system without it11, but the advantages of virtual memory far outweigh the disadvantages, especially when used for debugging purposes.

Node:Call stacks and symbol tables, Next:, Previous:Virtual memory, Up:Operating system support

6.2 Call stacks and symbol tables

As stated in the section on stack memory allocations (see Stack memory allocations), when a function is called, a copy of the caller's state information (including local variables and registers) is saved on the stack so that it can be restored when the called function returns. On many operating systems there is a calling convention12 which defines the layout of such stack entries so that code compiled in different languages and with different compilers can be intermixed. This usually specifies at which stack offsets the stack pointer, program counter and local variables for the calling function can be found, although on some processor architectures the function calling conventions are specified by the hardware and so the operating system must use these instead.

On systems that have consistent calling conventions, it is usually possible to perform call stack tracebacks from within the current function in order to determine the stack of function calls that led to the current function. This is extremely useful for debugging purposes and is done by examining the current stack frame to see if there is a pointer to the previous stack frame. If there is, then it can be followed to find out all of the state information about the calling function. This can be repeated until there are no more stack frames13. This is generally how this information is determined by debuggers when a call stack traceback is requested.

In addition to the pointer to the previous stack frame, the saved state information also always contains the saved program counter register, which contains either the address of the instruction that performed the function call, or the address of the instruction at which to continue execution when the called function returns14. This information can be used to identify which function performed the call, since the address of the instruction must lie between the start and end of one of the functions in the process.

There are several different ways to perform stack unwinding. The first requires compiler support and uses builtin functions to determine the next stack frame and the return address. The GNU C compiler, gcc, supports this but unfortunately the number of stack frames to traverse must be known at compile-time rather than run-time. The second method requires operating system support, with a library of routines provided to perform call stack traversal. Unfortunately, such routines can be quite time consuming and may require a lot of resources, but on the other hand they are likely to be very reliable at obtaining the necessary information. The mpatrol library can be built to support either of these methods, with the MP_BUILTINSTACK_SUPPORT and the MP_LIBRARYSTACK_SUPPORT preprocessor macros.

A third way to perform stack unwinding involves reading (or effectively disassembling) the instructions that are being executed in order to determine the size of the stack frame being used and the address of the instruction at which execution will resume when the function returns. This can also be quite a reliable method of obtaining call stack information but is only likely to be feasible on a processor architecture which has a very simple instruction set, such as a RISC15 architecture. MIPS processors are a good example of this.

The final method of stack unwinding requires that the frame pointer and return address are both stored on the stack whenever a new function is called. The chain of frame pointers can then be followed down the stack, and the return addresses can be read at a given offset from the frame pointers. This is usually possible with CISC16 processor architectures that have dedicated call instructions which automatically save such information on the stack, although some RISC processors also save these as well. However, inline functions and compiler optimisations can sometimes result in the frame pointer being omitted, usually resulting in an inability to walk the stack.

However, in order to determine this symbolic information, it must be possible to find out where the start and end addresses of all of the functions in the process are. This can usually only be read from object files, since they contain the symbol tables that were used by the linker to generate the final executable file for the program. The object file's symbol tables normally contain information about the start address, size, name and visibility of every symbol that was defined, but this depends on the format of the object file and if the symbol tables have been stripped from the final executable file.

If the object file was created by a compiler then it may also contain debugging information that was generated by the compiler for use with a debugger. Such information may include a mapping of code addresses to source lines17, and this information can be used by the mpatrol library to provide more meaningful information in call stack tracebacks.

On systems that support shared libraries, additional work must be done to determine the symbolic information for all of the functions which have been defined in them. The symbols for functions that are defined in shared libraries normally appear as undefined symbols in the executable file for the program and so must be searched in the system in order to get the necessary information. It is usually necessary to liaise with the dynamic linker18 on many systems.

Node:Threads, Previous:Call stacks and symbol tables, Up:Operating system support

6.3 Threads

On systems with virtual memory, such as UNIX and Windows, user programs are run as processes which have their own address space and resources. If a process needs to create sub-processes to perform other tasks it must call fork() or spawn() to create new processes, but these new processes do not share the same address space or resources as the parent process. If processes need to share memory they must either use a message passing interface or explicitly mark a range of memory as shareable.

Traditionally, this was not too much of a handicap as parallel processing was an expensive luxury and could only be made use of by the kernel of such systems. However, with the birth of fast processors and parallel programming, programs could be made to run more efficiently and faster on multi-processor systems by having more than one thread of control. This was achieved by allowing processes to have more than one program counter through which the processor could execute instructions, and if one thread of control stalled for a particular reason then another could continue without stalling the entire process.

Such multithreaded programs allow parallel programming and implicit shared memory between threads since all threads in a process share the same address space and resources. This is similar to operating systems that have no virtual memory, such as AmigaOS and Netware19, except that once a process terminates, all threads terminate as well and all of its resources are still reclaimed.

Multithreaded programming generally needs no compiler support, but does require some primitive operations to be supported by the operating system for a threads library to call. The functions that are available in the threads library provide the means for a process to create and destroy threads. There are currently several popular threads libraries available, although the POSIX threads standard remains the definitive implementation.

It is always important to remember when programming a multithreaded application that because all threads in a process share the same address space, measures must be taken to prevent threads reading and writing global data in a haphazard fashion. This can either be done by locking with semaphores and mutexes, or can be performed by using stack variables instead of global variables since every thread has its own local stack. Care must be taken to write re-entrant functions -- i.e. a function will give exactly the same result with one thread as it will with multiple threads running it at the same time.

The mpatrol library can be built as a thread-safe library with support for multi-threaded programs. When this library is linked with your program, only one thread at a time can allocate, reallocate or deallocate dynamic memory, or perform a memory operation via memcpy(), memset(), etc. This does not take full advantage of the potential concurrency in the library, but at least it will allow the debugging of multi-threaded programs.

The process of making the mpatrol library thread-safe was made more complicated by the fact that the mutexes protecting the library's data structures had to be recursive, since some of the functions that the library will call may call malloc() and free() or any other functions redefined by the library. If this was to happen with non-recursive mutexes then the recursive call would result in the thread attempting to lock a mutex that it already owned. However, implementing recursive mutexes was only half the problem.

The other problem with writing a thread-safe malloc library is that it must be initialised before the program becomes multi-threaded. If the library is initialised when there are multiple threads running then one thread may be attempting to initialise the mutexes whilst another thread may be attempting to lock an uninitialised mutex. Ideally, the best place to initialise the library would be at the start of main() but there is currently no way to do this other than getting users to explicitly plant calls to initialise the library in their code. This is not a very satisfactory solution if all we want to do is link in the replacement malloc library without any need for recompilation.

Fortunately, there are some ways to plant initialisation calls before main() is called, but they all have some drawbacks. The first way is to use a static file-scope constructor in C++, which will then initialise the mutexes and the library data structures before the code in main() is executed. However, on many systems this will require the final link to be performed by the C++ compiler that built the library. That may not be desirable or even possible in many cases. Unfortunately, this drawback appears in the second method, which involves using the GNU C compiler to compile the library. This compiler has an extension which allows functions to be specified as constructors which will be called before main(), but means that any program which is linked with the resulting library must be linked with the GNU C compiler driver. However, many systems are now GNU-based which would mean that this would happen anyway.

The final way of initialising the mutexes and the library data structures is to plant a call to the initialisation routines from a special section which the system will call before main() is called. This section is called the .init section on ELF-based platforms, but may exist in another form on other platforms too. This has the advantage that the system linker can be used to link the final program, but a possible disadvantage is that the library may be initialised too early, possibly before the environment or file streams have been set up. You may find that if one of the above methods does not work for you then perhaps another one will.

Node:Using mpatrol, Next:, Previous:Operating system support, Up:Top

7 Using mpatrol

This chapter contains a general description of all of the features of mpatrol and how to use them effectively. You'll also find a complete reference for mpatrol in the appendices, but you may wish to try out the examples (see Examples) and the tutorial (see Tutorial) before reading further.

Node:Library behaviour, Next:, Up:Using mpatrol

7.1 Library behaviour

Most of the behaviour of the mpatrol library can be controlled at run-time via options which are read from the MPATROL_OPTIONS environment variable. This prevents you having to recompile or relink each time you want to change a library setting, and so makes it really easy to try out different settings to locate a particular bug. You should know how to set the value of an environment variable on your system before you read on.

By default, the mpatrol library will attempt to determine the minimum required alignment for any generic memory allocation when it first initialises itself. This may be affected by the compiler and its settings when the library was built but it should normally reflect the minimum alignment required by the processor on your system. If you would prefer a larger (or perhaps even smaller) default alignment you may change it at run-time using the DEFALIGN option. The value you supply must be in bytes, must be a power of two, and should not be larger that the system page size. If you encounter bus errors due to misaligned memory accesses then you should increase this value.

On systems that have virtual memory the library will attempt to write-protect all of its internal structures when user code is being run. This ensures that it is nearly impossible for a program to corrupt any mpatrol library data. However, unprotecting and then protecting the structures at every library call has a slight overhead so you may prefer to disable this behaviour by using the NOPROTECT option. This has no effect on systems that have no virtual memory.

Usually it is desirable for many system library routines to be protected from being interrupted by certain signals since they may themselves be called from signal handlers. If this is not the case then it may be possible to interrupt the program from within such routines, perhaps causing problems if their global variables are left in an undefined state. As the mpatrol library replaces some of these system library routines it is also possible to specify that they are protected from certain interrupt signals using the SAFESIGNALS option. However, this can sometimes result in it being hard to interrupt the program from the keyboard if a lot of processor time is spent in mpatrol routines, which is why this behaviour is disabled by default20.

On UNIX platforms, the fork() function can cause problems if it is used to make a copy of the parent process without immediately calling one of the exec() family of functions. This is because the child process inherits all of the memory allocations of the parent process, but also inherits the log, profile and trace files as well. If both the parent and child processes make subsequent memory allocations there will be multiple entries with the same allocation indices written to the log, profile or trace files. This can be most confusing when processing these files afterwards! As a workaround, the mpatrol library will always check the current process identifier every time one of its functions is called if the CHECKFORK option is used and will open new log, profile or trace files if it has determined that the process has been forked. If the CHECKFORK option is not used then a call to __mp_reinit() should be added as the first function call in the child process in order to duplicate the behaviour of the CHECKFORK option.

On UNIX systems, the usual way for malloc libraries to allocate memory from the process heap is through the sbrk() system call. This allocates memory from a contiguous heap, but has the disadvantage in that other library functions may also allocate memory using the same function, thus creating holes in the heap. This is not a problem for mpatrol, but you may have a suspicion that your bug is due to a function from another library corrupting your data so you may wish to use the USEMMAP option. This is only available on systems that have the mmap() system call and allows mpatrol to allocate all of its memory from a part of the process heap that is non-contiguous (i.e. each call to mmap() may return a block of memory that is completely unrelated to that returned by the previous call). It may also be required on some systems in order for the mpatrol library to implement memory protection.

Beginning with release 1.3.3, the mpatrol library now allocates its internal memory in the opposite way to user memory on UNIX systems that support the mmap() system call. This means that by default, user memory is allocated with sbrk() whereas internal memory is allocated with mmap(), and this behaviour is reversed when the USEMMAP option is used. This was done to segregate user memory from internal memory, and was especially required for the mptrace command's graphical display.

The CHECK option allows you to specify that every time an mpatrol library function is called the library will automatically check the freed memory and overflow buffers of every memory allocation, although this can slow program execution down, especially if you suspect the error you are looking for occurs at the 1000th memory allocation, for example. You can therefore use the CHECK option to specify a range of memory allocations at which the mpatrol library will automatically check the freed memory and overflow buffers. All other allocations that fall outside this range will not be checked. You can also specify an optional frequency at which this checking should be performed. No such checking is performed by default in mpatrol release 1.4.0 and onwards -- you must specify CHECK=- to get the original behaviour.

On UNIX systems, the mpatrol library can also invoke the mpedit command to edit source files that show up in any warnings or error messages that it generates. This can only happen if the diagnostic message can be traced back to a specific source line in the program; in many cases this is not possible. If editing the files is not required, a context listing of the appropriate source line can be generated instead. The EDIT option specifies that files are to be edited and the LIST option specifies that a context listing is to be generated. These options are mutually exclusive.

If the mpatrol library that was built for your system supports reading symbolic information from a program's executable file, but it cannot locate the executable file, or you wish to specify an alternative, you can use the PROGFILE option to do this. All this does is instruct the mpatrol library to read symbols from this file instead. Note that on systems that support dynamic linking, the library can also read symbols from a dynamically linked executable file that has had its normal symbol table stripped.

Finally, a list of all of the recognised options in the mpatrol library can be displayed to the standard error file stream by using the HELP option. This will not affect the settings of the library in any way, so you should be able to use other options at the same time.

Node:Logging and tracing, Next:, Previous:Library behaviour, Up:Using mpatrol

7.2 Logging and tracing

If you would like to see a complete log of all of the memory allocations, reallocations and deallocations performed by your program, use the LOGALL option. This provides detailed tracing for each of the mpatrol library functions, and a full description of the format of such tracing is given in Example 1 (see Example 1). Alternatively, you may select one or more types of functions to be traced using the LOGALLOCS, LOGREALLOCS, LOGFREES and LOGMEMORY options if you feel that the log file is too large when LOGALL is used. By default all diagnostics from the mpatrol library get sent to mpatrol.log in the current directory, but this can be changed using the LOGFILE option. In fact, you can also specify a directory where all log files from the mpatrol library will get placed by setting the LOGDIR environment variable.

On systems that support it, every log entry also contains a call stack traceback that may also include the names of the symbols that appear on the call stack. If the object file access library that mpatrol was built with has support for reading line number tables from object files then the USEDEBUG option will also try to determine the file name and line number for each entry in the call stack, but only if the object files contain the relevant debugging information. This information will only be available before program termination and so any call stack tracebacks that appear after the library summary will not be displayed with their corresponding file name and line number. This option will also slow down program execution since a search through the line number tables will have to be made every time a call stack is displayed. Alternatively, the mpsym command may be used to process an mpatrol log file with a debugger in order to obtain symbol names and source level information for any call stacks.

Because the alloca(), strdupa() and strndupa() functions automatically free their allocations when the calling function returns, the log entries for these types of memory allocation are slightly different. The actual memory allocation will have an entry similar to malloc(), etc., but the memory deallocation will be marked as being done by alloca() and will occur at the next call to an mpatrol library function after the calling function has returned. However, any such allocations that are explicitly deallocated with the dealloca() function will be marked as being done by dealloca().

The mpatrol library will always try to display as much useful information as possible in this log file, and will always display a summary of library settings and statistics when your program terminates successfully. If you don't get this then your program did not call exit() and either called abort() or was terminated by the operating system instead. In such cases, either use a debugger to see where your program crashed or use the LOGALL option to see the last successful library call in the log file so that you have a rough idea of where your program crashed.

It is also possible to get mpatrol to write more summary information to the log file after it writes out its settings and statistics at program termination. Use the SHOWFREED and SHOWUNFREED options to display a list of freed and unfreed memory allocations. The former will only be displayed if the NOFREE option is used, but the latter can be useful for detecting memory leaks. The SHOWFREE option can be used to display a summary of any free memory blocks.

The SHOWMAP option will display a memory map of the heap that was valid when the process terminated, and the SHOWSYMBOLS option will display any symbolic information that the mpatrol library managed to obtain from any executable files and libraries that were relevant to the program being tested. All of the above five options can be selected with the SHOWALL option.

For the purpose of detecting memory leaks, you can instruct the mpatrol library to automatically log every memory allocation event into a special hash table called the leak table with the LEAKTABLE option. This option will then cause the mpatrol library to display a sorted summary of all of the memory leaks or unfreed memory allocations to the mpatrol log file when the program terminates. The leak table is indexed by the source file and line number where memory allocation events occur, but if this information is not available then either the function name or the return address will be used instead. Note that this option differs from the SHOWUNFREED option in that it will summarise where the leaks came from rather than show the full details of each individual unfreed memory allocation.

Because the log file can contain verbose information about memory allocations, reallocations, deallocations and operations, it can end up being too large if all such information is being logged for a large program. To get around this, it is possible to trace all memory allocation, reallocation and deallocation events in a concise way, to be stored in a separate file for later processing by the mptrace command. By default, no such tracing is performed but it can be enabled with the TRACE option. The default tracing output file is mpatrol.trace, but this can be changed using the TRACEFILE option. As with the LOGFILE option, you can also specify a directory where all tracing output files from the mpatrol library will get placed by setting the TRACEDIR environment variable.

Node:General errors, Next:, Previous:Logging and tracing, Up:Using mpatrol

7.3 General errors

By default, the mpatrol library follows the guidelines for ANSI C and C++ regarding the behaviour of the dynamic memory allocation and memory operation functions it replaces. This means that calling malloc() with a size of zero is allowed, for example. However, warnings can be generated for all of these types of calls by using the CHECKALL option. The CHECKALLOCS option warns only about calls to malloc() and similar functions with a size of zero, the CHECKREALLOCS option warns only about calls to realloc() and similar functions with either a null pointer or a size of zero, and the CHECKFREES option warns only about calls to free() and similar functions with a null pointer. The CHECKMEMORY option gives an error if a zero-size memory operation is performed on a NULL pointer -- this is normally allowed by default.

All newly-allocated memory can be pre-filled with a specified byte by using the ALLOCBYTE option. This can be used to catch out code that expects newly-allocated memory to be zeroed, although this option will have no effect on memory that was allocated with calloc(). All free memory can also be pre-filled with a different specified byte by using the FREEBYTE option. This will catch out code that expects to be able to use the contents of freed memory. Note that you may wish to change these options from their default values on your system so that the contents can be filled with values that are least likely to be used at run-time. For example, ensuring that the pointer representation of the value can never be a valid pointer, or that the floating point representation will always be invalid. These values will vary across operating systems and processor architectures.

Alternatively, the mpatrol library can be instructed to keep all (or a certain number of recent) freed memory allocations so that its diagnostics can be clearer about which freed allocation a piece of code is erroneously trying to access. This is controlled with the NOFREE option, which accepts an argument specifying the maximum number of recently-freed memory allocations to prevent being reused. If the argument is zero then all freed memory allocations will be immediately reused by the mpatrol library. If the argument is non-zero then the mpatrol library will use up more memory than usual since it has to keep all of the freed memory allocations lying around until their lifetime has expired. Note that this option distinguishes between free memory and freed memory. Free memory is unallocated memory that has been taken from the system heap. Freed memory is a freed memory allocation, with all of the original details of the allocation preserved.

Normally, the NOFREE option will fill the freed allocation with the free byte so that any code that accesses it will hopefully fall over. However, the original contents can be preserved using the PRESERVE option in case you need to see what the contents were just before it was freed. The NOFREE option is also affected by the PAGEALLOC option, since then the freed allocation will have its contents both read and write protected so that nothing can access them. If the PRESERVE option is used in this case then the freed allocation will only be made write-protected so that the original contents can be read from but not written to.

Note that if the argument specified with the NOFREE option is non-zero then the mpatrol library will store all recently-freed memory allocations in a queue. Once the queue has filled to the point specified with the NOFREE option then all subsequent calls to free memory will result in the most recently-freed memory allocation being placed at the end of the queue and the freed memory allocation at the beginning of the queue will be returned to the free memory pool for later reuse. Obviously, the larger the freed queue size, the better chance of detecting attempts to access previously-freed memory, but unfortunately more memory will be used up and the mpatrol library will have to keep track of a larger number of memory allocations.

Node:Overwrites and underwrites, Next:, Previous:General errors, Up:Using mpatrol

7.4 Overwrites and underwrites

Once a block of memory has been allocated, it is imperative that the program does not attempt to write any data past the end of the block or write any data just before the beginning of the block. Even writing a single byte just beyond the end of an allocation or just before the beginning of an allocation can cause havoc. This is because most malloc libraries store the details of the allocated block in the first few words before the beginning of the block, such as its size and a pointer to the next block. The mpatrol library does not do this, so a program which failed using the normal malloc library and worked when the mpatrol library was linked in is a possible candidate for turning on overflow buffers.

Such memory corruption can be extremely difficult to pinpoint as it is unlikely to show itself until the next call is made to the malloc library, or if the internal malloc library blocks were not overwritten, the next time the data is read from the block that was overwritten. If the former is the case then the next library call will cause an internal error or a crash, but only when the memory block that was affected is referenced. This is likely to disappear when using the mpatrol library since it keeps its internal structures separate, and write-protects them on systems that support memory protection.

In order to identify such errors, it is possible to place special buffers21 on either side of every memory allocation, and these will be pre-filled with a specified byte. Before every mpatrol library call, the library will check the integrity of every such overflow buffer in order to check for a memory underwrite or overwrite. Depending on the number of allocations and size of these buffers, this can take a noticable amount of time (which is why overflow buffers are disabled by default), but can mean that these errors get noticed sooner. The option which governs this is OFLOWSIZE. The byte with which they get pre-filled can be changed with OFLOWBYTE. Depending on what gets written, it might only be possible to see such errors when a different size of buffer or a different pre-fill byte is used.

Note that you may wish to change the OFLOWBYTE from its default value on your system so that the contents can be filled with values that are least likely to be used at run-time. For example, ensuring that the pointer representation of the value can never be a valid pointer, or that the floating point representation will always be invalid. These values will vary across operating systems and processor architectures, but may also vary depending on the datatypes that you will be expecting to store in the memory allocations.

A worse situation can occur when it is only reads from memory that overflow or underflow; i.e. with the faulty code reading just before or just past a memory allocation. These cannot be detected by overflow buffers as it is not possible using conventional means to interrupt every single read from memory. However, on systems with virtual memory, it is possible to use the memory protection feature to provide an alternative to overflow buffers, although at the added expense of increased memory usage.

The PAGEALLOC option turns on this feature and automatically rounds up the size of every memory allocation to a multiple of the system page size. It also rounds up the size of every overflow buffer to a multiple of the system page size so that every memory allocation occupies its own set of pages of virtual memory and no two memory allocations occupy the same page of virtual memory. The overflow buffers are then read and write protected so that any memory accesses to them will generate an error22. Following on from the previous section, the PAGEALLOC option also causes free memory to be read and write protected as well since that will also occupy non-overlapping virtual memory pages.

The remaining memory that is left over within an allocation's pages is effectively turned into traditional overflow buffers, being pre-filled with the overflow byte and checked periodically by the mpatrol library to ensure that nothing has written into them. However, because of this remaining memory, the library has a choice of where to place the memory allocation within its pages. If it places the allocation at the very beginning then it will catch memory underwrites, but if it places the allocation at the very end then it will catch memory overwrites. Such a choice can be controlled at run-time by supplying an argument to the PAGEALLOC option. If PAGEALLOC=LOWER is used then every allocation will be placed at the very beginning of its pages and if PAGEALLOC=UPPER is used then the placement will be at the very end of its pages. This is probably better explained in Example 3 (see Example 3) where the problems with PAGEALLOC=UPPER and alignment are also discussed.

Obviously, there are still some deficiencies when using PAGEALLOC since it can use up a huge amount of memory (especially with NOFREE) and the overflow buffers within an allocation's pages can still be read without causing an immediate error. Both of these deficiencies can be overcome by using the OFLOWWATCH option to install software watch points instead of overflow buffers, but there are still very few systems that support software watch points at the moment, and it can slow a program's execution speed down by a factor of around 10,000. The reason for this is that software watch points instruct the operating system to check every read from and write to memory, which means that it has to single-step through a process checking every instruction before it is executed. However, this is a very thorough way of checking for overflows and is unlikely to miss anything, although there may be problems with misaligned memory accesses when using watch points (see Virtual memory).

Note that from release 1.1.0 of mpatrol, the library comes with replacement functions for many memory operation functions, such as memset() and memcpy(). These new functions provide additional checks to ensure that if a memory operation is being performed on a memory block, the operation will not read or write before or beyond the boundaries of that block.

Normally, if an error is discovered in the call to such functions, the mpatrol library will report the error but prevent the operation from being performed before continuing execution. If the error was that the range of memory being operated on overflowed the boundaries of an existing memory allocation then the ALLOWOFLOW option can be used to turn the error into a warning and force the operation to continue. This behaviour can be desirable in certain cases where third-party libraries are being used that make such calls but the end result does not overflow the allocation boundary.

From release 1.3.3 of mpatrol, the library comes with functions that interface to the -fcheck-memory-usage option of the GNU compiler. This option instructs the compiler to place error-checking calls before each read or write to memory. The functions that are called then check to ensure that the memory access does not overflow a heap memory allocation or access free memory. This can be a very useful way to go through your code looking for errors with a fine tooth-comb, but be aware that it does slow down execution by a large factor. It also only affects functions that were compiled with this option, so if the problem lies in a function that was not recompiled with -fcheck-memory-usage then it won't do much good.

To conclude, if you suspect your program has a piece of code which is performing illegal memory underwrites or overwrites to a memory allocation you turn on the CHECK=- option and you should use each of the following options in sequence, but only if your system supports them. If all else fails and you are using the GNU compiler then you could try recompiling some or all of your code with the -fcheck-memory-usage option.

  1. OFLOWSIZE=8
  2. OFLOWSIZE=32
  3. OFLOWSIZE=1 PAGEALLOC=LOWER
  4. OFLOWSIZE=1 PAGEALLOC=UPPER
  5. OFLOWSIZE=8 OFLOWWATCH
  6. OFLOWSIZE=32 OFLOWWATCH

Node:Using with a debugger, Next:, Previous:Overwrites and underwrites, Up:Using mpatrol

7.5 Using with a debugger

If you would like to use mpatrol to pause at a specific memory allocation, reallocation or deallocation in a debugger then this section will describe how to go about it. Unfortunately, debuggers vary widely in function and usage and are normally very system-dependent. The example below will use gdb as the debugger, but as long as you know how to set a breakpoint within a debugger, any one will do.

First of all, decide where you would like the mpatrol library to pause when running your program within the debugger. You can choose one allocation index to break at using the ALLOCSTOP option, or you can choose to break at a specific reallocation of that allocation by also using the REALLOCSTOP option. If you use REALLOCSTOP without using ALLOCSTOP then you will break at the first memory allocation that has been reallocated the specified number of times. You can also choose to break at the point in your program that frees a specific allocation index by using the FREESTOP option.

The normal process for determining where you would like to pause your program in the debugger is by using the LOGALL option and examining the log file produced by mpatrol. If your program crashed then you should look at the last entry in the log file to see what the allocation index (and possibly also the reallocation index) of the last successful call was. You can then decide which of the above options to use. Note that the debugger will break at a point before any work is done by the mpatrol library for that allocation index so that you can see if it was the last successful operation that caused the damage.

Having decided which combination of mpatrol options to use, you should set them in the MPATROL_OPTIONS environment variable before running the debugger on your program. Alternatively, your debugger may have a command that allows you to modify your environment during debugging, but you're just as well setting the environment variable before you run the debugger as it shouldn't make any difference23.

After you get to the debugger command prompt, you should set a breakpoint at the __mp_trap() function. This is the function that gets called when the specified allocation index and/or reallocation index appears and so when you run your program under the debugger the mpatrol library will call __mp_trap() and the debugger will stop at that point. If you are not running your program within a debugger, or if you haven't set the breakpoint, then __mp_trap() will still be called, but it won't do anything. Note that there may be some naming issues on some platforms where the visible name of a global function gets an underscore prepended to it. You may have to take that into account when setting the breakpoint on such systems.

Now that you have set the MPATROL_OPTIONS environment variable and have set the debugger to break at __mp_trap(), all that is required is for you to run your program. Hopefully, the debugger should stop at __mp_trap(). If it doesn't then you may have to check your environment variable settings to ensure that they are the same as when you ran the program outwith the debugger, although obviously with the addition of ALLOCSTOP, etc. Once the program has been halted by the debugger, you can then single-step through your code until you see where it goes wrong. If this is near the end of your program then you'll have saved yourself a lot of time by using this method.

The following example will be used to illustrate the steps involved in using the ALLOCSTOP, REALLOCSTOP and FREESTOP options. However, it is only for tutorial purposes and the same effect could easily be achieved by breaking at line 18 in a debugger because in this case it is obvious from the code and the mpatrol log file where it is going wrong. In real programs this is hardly ever the case24. 

 1  /*
 2   * Allocates 1000 blocks of 16 bytes, freeing each block immediately
 3   * after it is allocated, and freeing the last block twice.
 4   */


 7  #include "mpatrol.h"


10  int main(void)
11  {
12      void *p;
13      int i;

15      for (i = 0; i < 1000; i++)
16          if (p = malloc(16))
17              free(p);
18      free(p);
19      return EXIT_SUCCESS;
20  }

Compile this example code with debugging information enabled and link it with the mpatrol library, then set MPATROL_OPTIONS to LOGALL and run the resulting program. If you examine mpatrol.log you will see the following near the bottom of the file. 

...

ALLOC: malloc (1000, 16 bytes, 4 bytes) [main|test.c|16]
        0x08049449 main+57
        0x4007C9CB __libc_start_main+255
        0x08049381 _start+33

returns 0x080620E8

FREE: free (0x080620E8) [main|test.c|17]
        0x08049470 main+96
        0x4007C9CB __libc_start_main+255
        0x08049381 _start+33

    0x080620E8 (16 bytes) {malloc:1000:0} [main|test.c|16]
        0x08049449 main+57
        0x4007C9CB __libc_start_main+255
        0x08049381 _start+33

FREE: free (0x080620E8) [main|test.c|18]
        0x08049491 main+129
        0x4007C9CB __libc_start_main+255
        0x08049381 _start+33

ERROR: [NOTALL]: free: 0x080620E8 has not been allocated

...

In this example, we'll want to use ALLOCSTOP to halt the program at the 1000th memory allocation so that we can step through it with a debugger. So, set MPATROL_OPTIONS to ALLOCSTOP=1000 and load the program into the debugger. If you are using gdb you can now do the following steps, but if you are not you will have to use the equivalent commands in your debugger. Note that (gdb) is the debugger command prompt and so anything that appears on that line after that should be typed as a command. 

(gdb) break __mp_trap
Breakpoint 1 at 0x804ee83
(gdb) run
Starting program: a.out
Breakpoint 1, 0x804ee83 in __mp_trap()
(gdb) backtrace
#0  0x804ee83 in __mp_trap()
#1  0x804c61b in __mp_getmemory()
#2  0x8049894 in __mp_alloc()
#3  0x8049449 in main() at test.c:16
(gdb) finish
Run till exit from #0  0x804ee83 in __mp_trap()
0x804c61b in __mp_getmemory()
(gdb) finish
Run till exit from #0  0x804c61b in __mp_getmemory()
0x8049894 in __mp_alloc()
(gdb) finish
Run till exit from #0  0x8049894 in __mp_alloc()
0x8049449 in main() at test.c:16
16              if (p = malloc(16))
(gdb) step
17                  free(p);
(gdb) step
15          for (i = 0; i < 1000; i++)
(gdb) step
18          free(p);
(gdb) quit
The program is running.  Exit anyway? (y or n) y

After setting the breakpoint and running the program, the debugger halts at __mp_trap(). Because __mp_trap() is a function within the mpatrol library, you don't want to bother stepping through any of the library functions, and in this case you can't since the mpatrol library was not compiled with debugging information enabled. So, after returning from all of the library functions, the source line becomes line 16 because that was the location of the 1000th memory allocation. Single-stepping twice gets us to line 18 which is our destination. Note that the file extra/.gdbinit included in the mpatrol distribution contains predefined commands which make setting the allocation index to stop at much easier.

Sometimes it is useful to be able to see information about a memory allocation whilst running a program from within a debugger. The __mp_printinfo() function is provided for that purpose and takes a heap address as its only argument. Using the above example, it would have been possible to print out information about the pointer p at line 17 from within gdb: 

(gdb) call __mp_printinfo(p)
address 0x080620E8 located in allocated block:
    start of block:     0x080620E8
    size of block:      16 bytes
    stored type:        <unknown>
    stored type size:   <unknown>
    user data:          0x00000000
    allocated by:       malloc
    allocation index:   1000
    reallocation index: 0
    modification event: 1999
    flags:              none
    calling function:   main
    called from file:   test.c
    called at line:     16
    function call stack:
        0x08049449 main
        0x4007C9CB __libc_start_main
        0x08049381 _start

Some debuggers, such as gdb, also allow you to define your own commands for use in a debugging session. The following example defines a new gdb command called printalloc which calls __mp_printinfo()25: 

(gdb) define printalloc
Type commands for definition of "printalloc".
End with a line saying just "end".
>call __mp_printinfo($arg0)
>end
(gdb) document printalloc
Type documentation for "printalloc".
End with a line saying just "end".
>Displays information about an address in the heap.
>end

Node:Testing, Next:, Previous:Using with a debugger, Up:Using mpatrol

7.6 Testing

The mpatrol library has several features that make it useful when testing a program's dynamic memory allocations. These are features that do not help in fixing an existing bug, but rather help to identify additional bugs that may be lurking in your code.

It is possible to set a simulated upper limit on the amount of heap memory available to a process with the LIMIT option, which accepts a size in bytes, but will be disabled when it is zero. This can be extremely useful for testing a program under simulated low memory conditions to see how it handles such errors. Of course, you should set the heap limit to a value less than the amount of actual available memory otherwise this option will have no effect. Note that the mpatrol library may use up a small amount of heap memory when it initialises itself26 so the value passed to the LIMIT option may need to be set slightly higher than you would normally expect.

It is also possible to instruct the mpatrol library to randomly fail a certain number of memory allocations so that you can further test error handling code in a program. The frequency at which failures occur can be controlled with the FAILFREQ option, where a value of zero means that no failures will occur, but any other value will randomly cause failures. For example, a value of 10 will cause roughly one in ten failures and a value of 1 will cause every memory allocation to fail. The random sequence can be made predictable by using the FAILSEED option. If this is non-zero then the same program run with the same failure frequency and same failure seed will fail on exactly the same memory allocations. If this is zero then the failure seed will itself be set randomly, but you can see its value when the summary is displayed at program termination.

When running batch tests27 it is sometimes useful to be able to detect if there have been any memory leaks. Such leaks should normally be distinguished from code which has purposely not freed the memory that it allocated, so there may be a certain expected number of unfreed allocations at program termination. It may be that you would like to highlight any additional unfreed allocations since they may be due to real memory leaks, so the UNFREEDABORT option can be set to a threshold number of expected unfreed allocations. If the library detects a number of unfreed allocations higher than this then it will abort the program at termination so that it fails. All tests that fail in this way can then be examined after the test suite finishes.

Node:Library functions, Next:, Previous:Testing, Up:Using mpatrol

7.7 Library functions

Along with the standard set of C and C++ dynamic memory allocation functions, the mpatrol library also comes with an additional set of functions which can be used to provide additional information to your program, and which can be called at various points in your code for debugging purposes. You must always include the mpatrol.h header file in order to use these functions, but you can check for a specific version of the mpatrol library by checking the MPATROL_VERSION preprocessor macro. You can check the version of the mpatrol library that a program was linked with by calling the __mp_libversion() function.

Certain mpatrol library options can be set after the library has been initialised with the __mp_setoption() function. This allows you to override the default options or those specified in the MPATROL_OPTIONS environment variable from within your code. Not all options can be overridden, however, since they would require a complete reinitialisation of the library -- the __mp_setoption() function returns a failure indicator in these cases. You can read the setting of any mpatrol library option with the corresponding function, __mp_getoption().

On systems that support it, global functions (with C linkage) in an executable file or shared library whose names begin with __mp_init_ will be noted when the mpatrol library first starts up and is reading the symbols. Such functions will then be called as soon as the mpatrol library is initialised, which can be useful if the initialisation occurs before main() is called. These functions must accept no arguments and must return no value. Similar behaviour exists for global functions whose names begin with __mp_fini_, except that such functions will be executed when the mpatrol library terminates. Note that this feature will have no effect if the symbol table is stripped from the executable file or shared library before the program is run, and the order in which such functions will be called if there are more than one is unspecified. The __mp_atexit() function can also be used to register functions that should be called when the mpatrol library terminates.

It is possible to obtain a great deal of information about an existing memory allocation or free block using the __mp_info() function. This takes an address as an argument and fills in any details about its corresponding memory allocation in a supplied structure. The following example illustrates this (it can be found in tests/pass/test4.c). 

 23  /*
 24   * Demonstrates and tests the facility for obtaining information
 25   * about the allocation a specific address belongs to.
 26   */


 29  #include "mpatrol.h"
 30  #include <stdio.h>


 33  void display(void *p)
 34  {
 35      __mp_allocstack *s;
 36      __mp_allocinfo d;
 37      __mp_symbolinfo i;

 39      if (!__mp_info(p, &d) || !d.allocated)
 40      {
 41          fprintf(stderr, "nothing known about address 0x%0*lX\n",
 42                  sizeof(void *) * 2, p);
 43          return;
 44      }
 45      fprintf(stderr, "block:    0x%0*lX\n", sizeof(void *) * 2, d.block);
 46      fprintf(stderr, "size:     %lu\n", d.size);
 47      fprintf(stderr, "type:     %s\n", __mp_function(d.type));
 48      fprintf(stderr, "alloc:    %lu\n", d.alloc);
 49      fprintf(stderr, "realloc:  %lu\n", d.realloc);
 50      fprintf(stderr, "thread:   %lu\n", d.thread);
 51      fprintf(stderr, "event:    %lu\n", d.event);
 52      fprintf(stderr, "func:     %s\n", d.func ? d.func : "<unknown>");
 53      fprintf(stderr, "file:     %s\n", d.file ? d.file : "<unknown>");
 54      fprintf(stderr, "line:     %lu\n", d.line);
 55      for (s = d.stack; s != NULL; s = s->next)
 56      {
 57          fprintf(stderr, "\t0x%0*lX", sizeof(void *) * 2, s->addr);
 58          if (__mp_syminfo(s->addr, &i))
 59          {
 60              if (i.name != NULL)
 61                  fprintf(stderr, " %s", i.name);
 62              if ((i.addr != NULL) && (i.addr != s->addr))
 63                  fprintf(stderr, "%+ld",
 64                          (char *) s->addr - (char *) i.addr);
 65              if (i.object != NULL)
 66                  fprintf(stderr, " [%s]", i.object);
 67          }
 68          else if (s->name != NULL)
 69              fprintf(stderr, " %s", s->name);
 70          fputc('\n', stderr);
 71      }
 72      fprintf(stderr, "typestr:  %s\n",
 73              d.typestr ? d.typestr : "<unknown>");
 74      fprintf(stderr, "typesize: %lu\n", d.typesize);
 75      fprintf(stderr, "userdata: 0x%0*lX\n", sizeof(void *) * 2, d.userdata);
 76      fputs("flags:   ", stderr);
 77      if (!d.freed && !d.marked && !d.profiled && !d.traced && !d.internal)
 78          fputs(" none\n", stderr);
 79      else
 80      {
 81          if (d.freed)
 82              fputs(" freed", stderr);
 83          if (d.marked)
 84              fputs(" marked", stderr);
 85          if (d.profiled)
 86              fputs(" profiled", stderr);
 87          if (d.traced)
 88              fputs(" traced", stderr);
 89          if (d.internal)
 90              fputs(" internal", stderr);
 91          fputc('\n', stderr);
 92      }
 93  }


 96  void func2(void)
 97  {
 98      void *p;

100      if (p = malloc(16))
101      {
102          display(p);
103          free(p);
104      }
105      display(p);
106  }


109  void func1(void)
110  {
111      func2();
112  }


115  int main(void)
116  {
117      func1();
118      return EXIT_SUCCESS;
119  }

When this is compiled and run, it should give the following output, although the pointers are likely to be different. 

block:    0x0806A0E8
size:     16
type:     malloc
alloc:    52
realloc:  0
thread:   0
event:    97
func:     func2
file:     test4.c
line:     100
          0x0804A743 func2+35 [./a.out]
          0x0804A790 func1+8 [./a.out]
          0x0804A79C main+8 [./a.out]
          0x4007C9CB __libc_start_main+255 [/lib/libc.so.6]
          0x0804A3E1 _start+33 [./a.out]
typestr:  <unknown>
typesize: 0
userdata: 0x00000000
flags:    none
nothing known about address 0x0806A0E8

As you can see, anything that the mpatrol library knows about any memory allocation can be obtained for use in your own code, which can be very useful if you need to write handlers to keep track of memory allocations, etc. for debugging purposes. It can also be useful to have this information when running your program within a debugger, so you can use the __mp_printinfo() function to display information about a heap address if your debugger supports calling functions from the command prompt. Note that the textual representation of the type field returned by the __mp_info() function can be obtained by calling __mp_function().

The mpatrol library records the error code from the most recently encountered warning or error in the __mp_errno global variable. This variable can be read and compared with the known error codes listed in mpatrol.h. It can also be reset to MP_ET_NONE before calling any mpatrol library function in order to check to see if a warning or error was encountered during the call. A string representation of the error message corresponding to any mpatrol error code can be obtained by calling the __mp_strerror() function with the specific code.

The userdata field shown in the previous example can be set for any memory allocation with the __mp_setuser() function. This can have any value and is not interpreted by the mpatrol library. It was added for user code to associate its own data with memory allocations.

The marked field that is also shown in the previous example indicates if a memory allocation has been marked to indicate that it should never be freed. This can only be performed from the source code by calling __mp_setmark() with the address of the memory allocation. Such a memory allocation can be reallocated but never freed, and will not contribute to the list of memory leaks. It will also be profiled and traced as freed by the end of program execution if memory allocation profiling or tracing is enabled.

You may also have noticed the use of __mp_syminfo() in the above example. This function is very similar to the __mp_info() function except that instead of looking for the details of a memory allocation at a specific address, it looks for the details of a function symbol at that address. This provides user access to the data obtained by the mpatrol symbol handler, including line number information if the USEDEBUG option is supported and used.

It is also possible for you to be able to intercept calls to allocate, reallocate and deallocate memory for your own purposes. You can install prologue and epilogue functions that the mpatrol library will call before and after every time one of its functions is called. These can be used for additional tracing or simply to add extra checks to your code. The following code is an example of this and can be found in tests/pass/test2.c. 

23  /*
24   * Demonstrates and tests the facility for specifying user-defined
25   * prologue and epilogue functions.
26   */


29  #include "mpatrol.h"
30  #include <stdio.h>


33  __mp_prologuehandler old_prologue;
34  __mp_epiloguehandler old_epilogue;


37  void prologue(MP_CONST void *p, size_t l, size_t m, MP_CONST char *s,
38                MP_CONST char *t, unsigned long u, MP_CONST void *a)
39  {
40      if (old_prologue != NULL)
41          old_prologue(p, l, m, s, t, u, a);
42      if (p == (void *) -1)
43          fprintf(stderr, "allocating %lu bytes\n", l);
44      else if (l == (size_t) -1)
45          fprintf(stderr, "freeing allocation 0x%0*lX\n", sizeof(void *) * 2, p);
46      else if (l == (size_t) -2)
47          fprintf(stderr, "duplicating string `%s'\n", p);
48      else
49          fprintf(stderr, "reallocating allocation 0x%0*lX to %lu bytes\n",
50                  sizeof(void *) * 2, p, l);
51  }


54  void epilogue(MP_CONST void *p, MP_CONST char *s, MP_CONST char *t,
55                unsigned long u, MP_CONST void *a)
56  {
57      if (p != (void *) -1)
58          fprintf(stderr, "allocation returns 0x%0*lX\n", sizeof(void *) * 2, p);
59      if (old_epilogue != NULL)
60          old_epilogue(p, s, t, u, a);
61  }


64  int main(void)
65  {
66      void *p, *q;

68      old_prologue = __mp_prologue(prologue);
69      old_epilogue = __mp_epilogue(epilogue);
70      if (p = malloc(16))
71          if (q = realloc(p, 32))
72              free(q);
73          else
74              free(p);
75      if (p = (char *) strdup("test"))
76          free(p);
77      __mp_prologue(old_prologue);
78      __mp_epilogue(old_epilogue);
79      return EXIT_SUCCESS;
80  }

Once again, if you compile and run the above code, you should see the following output. 

allocating 16 bytes
allocation returns 0x0806A0E8
reallocating allocation 0x0806A0E8 to 32 bytes
allocation returns 0x0806A0E8
freeing allocation 0x0806A0E8
duplicating string `test'
allocation returns 0x0806A0E5
freeing allocation 0x0806A0E5

Note that in the above code, the previous prologue and epilogue functions were recorded and called. If this is not done then your prologue and epilogue functions will completely override all others, which is not usually the expected behaviour. In case you're wondering what the last four arguments of the prologue and epilogue handlers are, they are the function name, file name, line number and call address of the function that called malloc() or a related function. These can be used in the handlers to see where they were called from.

Along with being able to install prologue and epilogue functions, you can also install a low-memory handler with the __mp_nomemory() function, which will be called by the mpatrol library if it ever runs out of memory during the call to a memory allocation function. This gives you the opportunity to use that handler to either free up any unneeded memory or simply to abort, thus removing the need to check for failed allocations. Note that the low-memory handler also accepts the same four common arguments that the prologue and epilogue handlers do.

It is also possible to iterate over all of the allocated and freed memory allocations that are currently in the heap at any point in a program. This is done by invoking the __mp_iterate() function with a callback function which is called once per allocation with the start address of the memory block being passed as the argument to the callback function. Any further information about the memory allocation can then be obtained via the __mp_info() function. Note that the __mp_iterateall() function does the same as the __mp_iterate() function except that it also includes all free memory blocks and memory allocations that are internal to the mpatrol library.

Differences in the heap allocations (their details, not their contents) between a previous point in a program's execution and the current point of execution can be determined by calling the __mp_snapshot() function and then invoking __mp_iterate() with that snapshot value as its second argument at a later point in execution. The callback function passed to __mp_iterate() will then only be invoked with the start address of any memory allocation that has been allocated or reallocated (or freed if the NOFREE option is being used) since the snapshot point. This makes it possible to detect localised memory leaks very easily, as the following example (found in tests/pass/test10.c) shows. 

 23  /*
 24   * Demonstrates and tests the facility for obtaining information on
 25   * local memory leaks.  Will also edit or list the location of each
 26   * leak if the EDIT or LIST option is in effect.
 27   */


 30  #include "mpatrol.h"
 31  #include <stdio.h>


 34  int callback(MP_CONST void *p, void *t)
 35  {
 36      __mp_allocstack *s;
 37      __mp_allocinfo d;

 39      if (!__mp_info(p, &d) || !d.allocated)
 40      {
 41          fprintf(stderr, "nothing known about address 0x%0*lX\n",
 42                  sizeof(void *) * 2, p);
 43          return -1;
 44      }
 45      if (!d.freed)
 46      {
 47          fprintf(stderr, "0x%0*lX", sizeof(void *) * 2, d.block);
 48          fprintf(stderr, " %s", d.func ? d.func : "<unknown>");
 49          fprintf(stderr, " %s", d.file ? d.file : "<unknown>");
 50          fprintf(stderr, " %lu", d.line);
 51          for (s = d.stack; s != NULL; s = s->next)
 52          {
 53              if (s == d.stack)
 54                  fputs(" (", stderr);
 55              else
 56                  fputs("->", stderr);
 57              if (s->name != NULL)
 58                  fprintf(stderr, "%s", s->name);
 59              else
 60                  fprintf(stderr, "0x%0*lX", sizeof(void *) * 2, s->addr);
 61              if (s->next == NULL)
 62                  fputc(')', stderr);
 63          }
 64          fputc('\n', stderr);
 65          if ((d.file != NULL) && (d.line != 0))
 66              __mp_view(d.file, d.line);
 67          *((unsigned long *) t) = *((unsigned long *) t) + d.size;
 68          return 1;
 69      }
 70      return 0;
 71  }


 74  void func2(unsigned long n)
 75  {
 76      void *p;

 78      p = malloc((n * 10) + 1);
 79      if (n % 13)
 80          free(p);
 81  }


 84  void func1(void)
 85  {
 86      void *p;
 87      size_t i, n;
 88      unsigned long s, t;

 90      p = malloc(16);
 91      s = __mp_snapshot();
 92      for (i = 0; i < 128; i++)
 93          func2(i);
 94      free(p);
 95      t = 0;
 96      if (n = __mp_iterate(callback, &t, s))
 97          fprintf(stderr, "Detected %lu memory leaks (%lu bytes)\n", n, t);
 98      if ((n != 10) || (t != 5860))
 99          fputs("Expected 10 memory leaks (5860 bytes)\n", stderr);
100  }


103  int main(void)
104  {
105      void *p;

107      p = malloc(16);
108      func1();
109      free(p);
110      return EXIT_SUCCESS;
111  }

Compiling this example with mpatrol and then running it will produce the following list of memory leaks that were located between the points of calling __mp_snapshot() and __mp_iterate(). 

0x0806A108 func2 test10.c 78 (func2->func1->main->_start)
0x0806A674 func2 test10.c 78 (func2->func1->main->_start)
0x0806A6F8 func2 test10.c 78 (func2->func1->main->_start)
0x0806A800 func2 test10.c 78 (func2->func1->main->_start)
0x0806A988 func2 test10.c 78 (func2->func1->main->_start)
0x0806AB94 func2 test10.c 78 (func2->func1->main->_start)
0x0806AE20 func2 test10.c 78 (func2->func1->main->_start)
0x0806B130 func2 test10.c 78 (func2->func1->main->_start)
0x0806B4C0 func2 test10.c 78 (func2->func1->main->_start)
0x0806B8D4 func2 test10.c 78 (func2->func1->main->_start)
Detected 10 memory leaks (5860 bytes)

The tools directory in the mpatrol distribution contains two files called heapdiff.c and heapdiff.h which demonstrate the use of __mp_snapshot() and __mp_iterate() to find localised memory leaks. Have a look at these files to see a further example of using these functions, or perhaps even add these files to your application for debugging purposes. Note that it is perfectly safe to allocate memory in the callback function used by __mp_iterate(), and such allocations can be freed as well. The only restriction is that the callback function should never free a memory allocation that it has not allocated itself.

An alternative way to detect differences in the heap between two points in a program's execution is to make use of the leak table. This is a hash table that stores the number and size of memory allocations and deallocations referenced by the source file and line number where they occur28. The leak table can be cleared with a call to __mp_clearleaktable() and can be displayed with a call to __mp_leaktable(), which will display a sorted summary of the allocated, freed or unfreed memory entries stored in the leak table. Memory allocation events can be automatically logged in the leak table by calling __mp_startleaktable() but this behaviour can be disabled by calling __mp_stopleaktable(). Additional entries can be manually added to the leak table with __mp_addallocentry() and __mp_addfreeentry().

If you wish to write your own diagnostics to the mpatrol log file from within your source code then you can do so with the __mp_printf() and __mp_vprintf() functions, which are the functional equivalents of printf() and vprintf(). They prefix every line written to the log file with >, partly for making it clear where user diagnostics occur and partly to avoid problems with external utilities that parse the mpatrol log file. The __mp_locprintf() and __mp_vlocprintf() functions are equivalent functions that also display the source file and line number from where they were called along with a stack trace, if available.

It is also possible to write out a memory dump in hexadecimal format, a stack trace at the current point in execution and details of a memory allocation to the log file in standard format using the __mp_logmemory(), __mp_logstack() and __mp_logaddr() functions respectively.

You can also take advantage of the mpedit command from within the mpatrol library with the __mp_edit(), __mp_list() and __mp_view() functions. The first invokes a text editor on a specified file and line number, while the second displays a context listing of a file at a given line number. The third function performs either or neither depending on the setting of the EDIT or LIST options.

Finally, there are four functions which affect the mpatrol library globally. The first, __mp_check(), allows you to force an internal check of the mpatrol library's data structures at any point during program execution and also to free up any out of scope memory allocations made by the alloca() family of functions. The __mp_memorymap() function allows you to force the generation of a memory map at any point in your program, in much the same way as it would normally be displayed at the end of program execution if the SHOWMAP option was used. The __mp_summary() function writes library statistics to the mpatrol log file, while the __mp_stats() function fills in a data structure with selected statistics for examination in user code.

Node:Leak table, Previous:Library functions, Up:Using mpatrol

7.8 Leak table

The mpatrol library provides a hash table called a leak table that can be used to record memory allocations and deallocations for the purpose of detecting memory leaks. It can be fully controlled from the source code of a program by calling the appropriate mpatrol library functions, but the mpatrol library can also be instructed to automatically enter the details of each memory event into the leak table by using the LEAKTABLE option.

The leak table records a flat profile of memory allocation behaviour between two points in a program and is keyed by source file location. What that means is, it contains an entry for each source file and line number that allocated memory, and if more than one memory allocation event occurred at that point then the entry will summarise the total events that occurred at that point. In many cases, the source file and line number is not available for a memory allocation event, in which case either the function name or the return address can be used instead.

The following example shows the use of the leak table manipulation functions. 

 1  /*
 2   * Illustrates the explicit and implicit manipulation of the
 3   * leak table.
 4   */


 7  #include "mpatrol.h"


10  int main(void)
11  {
12      void *p, *q;
13      int r;

15      __mp_clearleaktable();
16      __mp_addallocentry("file.c", 1, 1);
17      __mp_addfreeentry("file.c", 1, 1);
18      __mp_addallocentry("file.c", 1, 2);
19      __mp_addfreeentry("file.c", 1, 2);
20      __mp_addallocentry("file.c", 1, 3);
21      __mp_addfreeentry("file.c", 1, 3);
22      __mp_addallocentry("function", 0, 8);
23      __mp_addfreeentry("function", 0, 4);
24      __mp_addallocentry("function", 0, 16);
25      __mp_addfreeentry("function", 0, 12);
26      __mp_addallocentry(NULL, 0x40000000, 8);
27      r = __mp_startleaktable();
28      if (p = malloc(16))
29          if (q = realloc(p, 32))
30              free(q);
31          else
32              free(p);
33      if (r == 0)
34          __mp_stopleaktable();
35      __mp_leaktable(0, MP_LT_ALLOCATED, MP_LT_BOTTOM);
36      __mp_printf("\n");
37      __mp_leaktable(0, MP_LT_FREED, MP_LT_COUNTS);
38      __mp_printf("\n");
39      __mp_leaktable(0, MP_LT_UNFREED, 0);
40      __mp_printf("\n");
41      return EXIT_SUCCESS;
42  }

The output that appears in mpatrol.log should look similar to this. 

bottom 5 allocated memory entries in leak table:

       bytes   count  location
    --------  ------  --------
           6       3  file.c line 1
           8       1  0x40000000
          16       1  test.c line 28
          24       2  function
          32       1  test.c line 29
          86       8  total

top 4 freed memory entries in leak table:

     count     bytes  location
    ------  --------  --------
         3         6  file.c line 1
         2        16  function
         1        32  test.c line 29
         1        16  test.c line 28
         7        70  total

top 2 unfreed memory entries in leak table:

       bytes   count  location
    --------  ------  --------
           8       1  0x40000000
           8       0  function
          16       1  total

Line 15 of the above program clears the leak table. This can be done as many times as necessary during the execution of the program, but note that there is only one leak table so this will affect the LEAKTABLE option. The following 6 lines then add matching allocation and deallocation entries to the leak table corresponding to a source location of line 1 in a file called file.c. Any deallocation entries must match the source position of the original allocation due to the nature of the leak table. Note that these 6 lines correspond to an allocation of 1 byte, followed by two reallocations of 2 and 3 bytes respectively, followed by a final deallocation.

Lines 22-23 allocate 8 bytes from a function called function() but only free 4 bytes, resulting in a memory leak of 4 bytes. Lines 24-25 do the same but allocate 16 bytes and free 12. The next line allocates 8 bytes from a code address of 0x40000000 but does not free it, resulting in a memory leak of 8 bytes. Note that if the line number is specified as 0 then the filename will be taken as a function name, and if the filename is specified as NULL then the line number will be taken as a return address. If neither are specified then the number of bytes will be added or subtracted from the unknown location.

Line 27 instructs mpatrol to automatically record any subsequent memory allocation events in the leak table, while line 34 does the opposite. The __mp_startleaktable() function returns 1 if automatic leak table recording was turned on and 0 otherwise, so we check in line 33 to see if we should turn off automatic recording by examining the previous state. Lines 28-32 allocate, reallocate and free some memory just to demonstrate this.

Finally, lines 35-40 display the contents of the memory leak table that have been recorded since the call to __mp_clearleaktable(). Three different types of display can be generated: allocated memory allocations, freed memory allocations and unfreed memory allocations. They can also be sorted by the total number of calls instead of the total number of bytes (MP_LT_COUNTS) or in reverse order with the smallest first (MP_LT_BOTTOM). The count of 0 in the last table indicates that there were a matching number of calls to allocate and free memory within function() but the number of bytes freed was less than the number of bytes allocated. This can only happen if the entries were added by __mp_addallocentry() and __mp_addfreeentry() and is usually an indication that something is wrong when making these calls.

The leak table is a useful tool to manipulate when debugging your application to check for memory leaks, but it is not as detailed as full memory allocation profiling which is controlled by the PROF option. It is probably better at summarising unfreed allocations than the SHOWUNFREED option but does not show the call stack for each allocation.

In addition, the __mp_iterate() function can be used within user code to obtain details about changes in the heap since a certain point in a program's execution. However, it can only provide details about freed memory allocations if the NOFREE option is used. The leak table will provide less details on freed memory allocations but its advantage is that it does not require the use of the NOFREE option.

Node:Tools, Next:, Previous:Using mpatrol, Up:Top

8 Tools

The tools directory that comes with the mpatrol distribution contains the source code for tools that are built on top of the mpatrol library. The functions that are defined in these files are intended to be useful for specific applications as well as providing real-world examples of how to extend mpatrol. If you wish to use one of the source files in the tools directory then you should first compile it and then link it into your program along with the mpatrol library.

Alternatively, if you've already installed mpatrol on your system then there should be an mpatrol subdirectory within the include directory where mpatrol.h is installed that contains all of the header files in the tools directory. There should also be a libmptools library within the library directory where libmpatrol is installed that contains an object file for each of the source files in the tools directory. You can then make use of a particular tool by including its header file from the mpatrol include subdirectory and then linking with the libmptools library.

If you've written a useful extension to mpatrol then you might wish to submit it for inclusion in the next release of mpatrol. Even if it's just for a specific application, there might be other users out there that may benefit from it. You'll even get a credit in the manual! Note that any documentation should also be written in the associated header file.

Node:dbmalloc, Next:, Up:Tools

8.1 Dbmalloc-compatible functions

This file provides Dbmalloc29-compatible functions which are built on top of the mpatrol library. They are compatible with the last known public release of Dbmalloc (patch level 14), but only the Dbmalloc-specific functions are defined here, leaving the overriding of standard functions up to the mpatrol library. As the mpatrol library does not currently override the C library string functions and the X toolkit heap allocation functions, neither does this file.

The dbmallopt() function does not support the setting of all of the Dbmalloc options. In fact, most of them do not make sense when applied to the mpatrol library. Some of them have slightly changed behaviour due to the mapping process and some of them cannot be implemented due to the mpatrol library having been initialised beforehand.

The malloc_dump() function does not support the full recognition of the MALLOC_DETAIL option in that it does not display the additional columns and summary that the Dbmalloc library does. This is because this would make no sense when applied to the mpatrol library, but it does still affect whether freed allocations are shown in the listing (although the details for such allocations are slightly different, and there are no entries displayed for free memory blocks).

The output for the malloc_dump() and malloc_list() functions is almost identical to that of the Dbmalloc library except for a slight change in the pointer format when displaying the address of each memory allocation. The stack information is obtained differently as well, since the mpatrol library records symbolic stack tracebacks for each memory allocation. As a result, malloc_enter() and malloc_leave() do nothing and the return address in a stack frame is displayed if no associated symbol name, file and line number could be determined. Parentheses are not printed at the end of symbol names so that they can be processed properly by a C++ demangler if necessary. Passing a file descriptor of 0 to malloc_dump() or malloc_list() results in the output being sent to the mpatrol log file.

The malloc_size() and malloc_mark() functions do not give an error message if the pointer passed in does not correspond to a heap allocation. Neither of these functions automatically perform an integrity check of the heap. Note that the malloc_chain_check() function will never return a non-zero value -- it always terminates with an error message in the mpatrol log file whenever it detects heap corruption. As a result, the malloc_abort() function is not used.

This file is initialised via the mpatrol library's initialiser function feature, which means that if the __mp_init_dbmalloc() function is noted by the mpatrol symbol manager then it will be called when the mpatrol library is being initialised. If this feature is not supported then the dbmallinit() function must be called as early on as possible, otherwise this file will not be initialised until one of its functions are called.

Node:dmalloc, Next:, Previous:dbmalloc, Up:Tools

8.2 Dmalloc-compatible functions

This file provides Dmalloc30-compatible functions which are built on top of the mpatrol library. They are compatible with the 4.8.2 release of Dmalloc, but only the Dmalloc-specific functions are defined here, leaving the overriding of standard functions up to the mpatrol library. As the mpatrol library does not currently override the C library string functions, neither does this file. In addition, the Dmalloc distribution comes with definitions for xmemalign(), xvalloc() and xrecalloc(), neither of which are defined by this file or by the mpatrol library.

This module is intended to work with the existing dmalloc command, which sets the contents of the DMALLOC_OPTIONS environment variable according to any specified command line options. The four documented Dmalloc global variables are also defined, although the two address variables are not acted upon and changing the dmalloc_logpath variable has no effect yet. The dmalloc_errno variable is mapped onto the __mp_errno variable and so the dmalloc_strerror() function always returns strings that are specific to the mpatrol library. Note that unlike the actual Dmalloc library, this file is not threadsafe, and the lockon option has no effect. In addition, the start option ignores the file:line syntax and uses allocation indices rather than events.

The dmalloc_debug() function does not support the setting of all of the Dmalloc flags, although this file defines preprocessor macros for each of them, something which is not currently done in the dmalloc.h file in the Dmalloc distribution. In fact, many of them do not make sense when applied to the mpatrol library. Some of them have slightly changed behaviour due to the mapping process and some of them cannot be implemented due to the mpatrol library having been initialised beforehand.

The dmalloc_verify() and dmalloc_examine() functions do not give an error message if the pointer passed in does not correspond to a heap allocation, and the latter function does not automatically perform an integrity check of the heap. The malloc_verify() function has not been included in this implementation since it is functionally identical to dmalloc_verify(). Note that the dmalloc_verify() function will only ever return DMALLOC_VERIFY_ERROR if the pointer to be checked is not null and is invalid -- it always terminates with an error message in the mpatrol log file whenever the pointer to be checked is null and it has detected heap corruption.

The dmalloc_log_heap_map() and dmalloc_log_stats() functions map on to the __mp_memorymap() and __mp_summary() functions and so have entirely different display formats. The dmalloc_log_unfreed() and dmalloc_log_changed() functions have similar display formats to the original Dmalloc library, but the summary tables are displayed differently and will display symbol names if they are available and filename and line number information isn't. The dmalloc_message() and dmalloc_vmessage() functions write tracing to the mpatrol log file prefixed by three fields of optional information, which can be controlled by the LOG_* macros when building this module.

This file is initialised via the mpatrol library's initialiser function feature, which means that if the __mp_init_dmalloc() function is noted by the mpatrol symbol manager then it will be called when the mpatrol library is being initialised. If this feature is not supported then the dmalloc_init() function must be called as early on as possible, otherwise this file will not be initialised until one of its functions are called.

Node:heapdiff, Next:, Previous:dmalloc, Up:Tools

8.3 Determining heap differences

This file defines heapdiffstart() and heapdiffend(), which must be called in matching pairs. They both take a heapdiff object as their first parameter, which must still be in scope when the matching call to heapdiffend() is made. The heapdiff object is initialised at the call to heapdiffstart() and is finalised when heapdiffend() is called. It must not be modified in between and should be treated as an opaque type. heapdiffend() can only be called once per heapdiff object before requiring that the heapdiff object be reinitialised through a call to heapdiffstart().

The second parameter to heapdiffstart() specifies a set of flags that can be used to control what is written to the mpatrol log. A list of all unfreed memory allocations can be logged with the HD_UNFREED flag and a list of all freed memory allocations can be logged with the HD_FREED flag, although the latter makes use of the NOFREE option and can incur a large performance and space penalty, and also relies on the NOFREE option being unmodified between the calls to heapdiffstart() and heapdiffend(). Note that marked allocations are not normally logged but this can be changed by adding the HD_MARKED flag.

By default, only a minimal amount of detail is logged for each allocation, but this can be changed with the HD_FULL flag to log full details for each allocation. If the filename and line number for an allocation is known and the EDIT or LIST option is being used then using HD_VIEW will edit or list the relevant source file at the correct line number, but only if the EDIT or LIST options are supported.

If the HD_CONTENTS flag is specified then the contents of all current memory allocations will be written to files and then compared with their subsequent contents when heapdiffend() is called. If the heap is large then this option can require a substantial amount of disk space. All of the allocation contents files will be deleted when the matching call to heapdiffend() is made.

Node:mgauge, Next:, Previous:heapdiff, Up:Tools

8.4 Memory allocation gauge

This file defines mgaugestart(), mgaugeend(), mgaugeon() and mgaugeoff() which produce and control a simple memory allocation gauge in a terminal window. The gauge is displayed in textual form using the standard I/O library rather than using a graphics library. Since it is updated in real-time, it makes no sense to send the output of the gauge to a file. Only one gauge can be in use at any one time.

The first argument to mgaugestart() is the filename of the file to write the gauge to. As mentioned before, this should be a terminal file that can be displayed in real-time, such as /dev/pts* on UNIX systems or CON:#? on AmigaOS. If it is a null pointer then the standard error file stream will be used.

The second argument to mgaugestart() specifies the character that will be used to represent allocated memory. If this is given as whitespace, | or + then # will be used instead. The third argument specifies the number of bytes that the gauge represents. If the total allocated memory exceeds this then + will be appended to the gauge. The final argument specifies the frequency of memory allocation events at which the gauge should be updated. If it is specified as zero then all events will cause the gauge to be updated.

Node:mtrace, Previous:mgauge, Up:Tools

8.5 Memory allocation tracing

This file defines mtrace() and muntrace(), two functions which enable and disable memory allocation tracing respectively. These should be called in matching pairs but will have no effect unless the MALLOC_TRACE environment variable is set to the filename of the trace file to use. The resulting trace files can be processed by the mtrace perl script which is distributed with the GNU C library.

Node:Utilities, Next:, Previous:Tools, Up:Top

9 Utilities

Several external programs are supplied with the mpatrol distribution in the form of commands that can be used to enhance the functionality of the mpatrol library. Each command comes with its own UNIX manual page (although they also support the --help and --version options), but a few of the commands are written as UNIX shell scripts and so will not work on non-UNIX platforms. Note that the mprof command is documented in the profiling chapter (see Profiling) and the mptrace command is documented in the tracing chapter (see Tracing).

Node:The mpatrol command, Next:, Up:Utilities

9.1 The mpatrol command

A command is provided with the mpatrol distribution which can run programs that have been linked with the mpatrol library, using a combination of mpatrol options that can be set via the command line. Most of these options map directly onto their equivalent environment variable settings and exist mainly so that the user does not have to manually change the MPATROL_OPTIONS environment variable.

The main option that is the exception to this is the --dynamic option, which can be used to run a program under the control of the mpatrol library, even if it wasn't originally linked with the mpatrol library. This can only be done on systems that support dynamic linking and where the dynamic linker recognises the LD_PRELOAD or _RLD_LIST environment variables. Even then, it can only be used when the program that is being run has been dynamically linked with the system C library, rather than statically linked.

The reason for all of these limitations is that some SVR4 UNIX platforms have a special feature in the dynamic linker which can be told to override the symbols from one shared library using the symbols from another shared library at run-time. In this case, it involves replacing the symbols for malloc(), etc., in the system C library with the mpatrol versions, but only if they were marked as undefined in the original executable file and would therefore have to have been loaded from libc.so.

However, if a program qualifies for use with the --dynamic option, it means that you can trace all of its dynamic memory allocations as well as running it with any of the mpatrol library's debugging options. This is mainly a toy feature which allows you to view and manipulate the dynamic memory allocations of programs that you don't have the source for, but in theory it could be quite useful if you need to debug a previously released executable and are unable to recompile or relink it. Note that if the program being run is multithreaded then you must add the --threads option as well.

Note that the mpatrol command must be set up to use the correct object file format access libraries that are required for your system if you wish to use the --dynamic option. If the mpatrol library was built with FORMAT=FORMAT_COFF or FORMAT=FORMAT_XCOFF support then it must be told to preload the COFF access library (normally libld.so). If it was built with FORMAT=FORMAT_ELF32 or FORMAT=FORMAT_ELF64 support then it must be told to preload the ELF access library (normally libelf.so)31. If it was built with FORMAT=FORMAT_BFD support then it must be told to preload the GNU BFD access libraries (normally libbfd.so and libiberty.so)32. However, if these libraries only exist on your system in archive form then you must build libmpatrol.so with these extra libraries incorporated into it so that there are no dependencies on them at run-time. However, there may well be problems if the resulting shared library contains position-dependent code from the archive libraries you incorporated. The only way to find out is for you to try it and see.

If you have access to the GNU linker on your system then there may be a way to convert archive libraries into shared libraries if position-independent code is not necessarily required for building shared libraries on your system. If you use the --whole-archive and --shared linker options then the GNU linker will read the entire contents of one or more archive libraries before writing out a shared library. All going well, you should be able to use the new shared library in conjunction with the --dynamic mpatrol option.

In order to build a shared version of the mpatrol library with embedded object file format access libraries, you must first modify the Makefile you would normally use to build the mpatrol library. At the lines where the linker is invoked to build the shared library, you must explicitly add any object file format access libraries that you want to use at the end of the linker command line. This ensures that all references to such libraries will be resolved at link time rather than run time. You must then edit the file src/config.h and remove all of the libraries that you embedded from the definitions of the MP_PRELOAD_LIBS and MP_PRELOADMT_LIBS preprocessor macros. Finally, rebuild the shared version of the mpatrol library and the mpatrol command and see if your efforts were worth it.

Because the mpatrol command sets the MPATROL_OPTIONS environment variable for each of the programs it runs, it does not affect the value of the environment variable for the current process (except on AmigaOS and Netware where all processes share the same environment). However, if you wish to use the mpatrol command to set MPATROL_OPTIONS in the current process environment then you can use its --show-env option to help you do so. This option will apply all of the mpatrol command line options to the MPATROL_OPTIONS environment variable and then display its value on the standard output without actually running any programs. You can then manually set the environment variable with the output from the mpatrol command.

If you wish the MPATROL_OPTIONS environment variable to be set in the current shell process automatically with the mpatrol command then you must use some shell trickery. The following script excerpts can be found in extra/.profile, extra/.cshrc and extra/.gdbinit and can be inserted into your ksh/bash, csh/tcsh and gdb configuration files respectively. They each provide the mallopt command, which takes mpatrol command options and sets the MPATROL_OPTIONS environment variable in the current shell or debugger process. 

# mallopt for ksh/bash

function mallopt()
{
    export MPATROL_OPTIONS=`mpatrol --show-env "$@"`
    echo "$MPATROL_OPTIONS"
}

# mallopt for csh/tcsh

alias mallopt 'setenv MPATROL_OPTIONS "`mpatrol --show-env \!*`";
               echo "$MPATROL_OPTIONS"'

# mallopt for gdb

define mallopt
printf "Enter mpatrol library options: "
shell read arg; echo set environment MPATROL_OPTIONS
                `mpatrol --show-env $arg` >/tmp/mpatrol.gdb
source /tmp/mpatrol.gdb
shell rm -f /tmp/mpatrol.gdb
show environment MPATROL_OPTIONS
end
document mallopt
Sets mpatrol library options in the current process environment.
end

Node:The mleak command, Next:, Previous:The mpatrol command, Up:Utilities

9.2 The mleak command

Another utility program that is provided is called mleak and is useful for detecting memory leaks in log files produced by the mpatrol library. This program should be used if the mpatrol library could not finish writing the log file due to abnormal program termination (which would prevent the SHOWUNFREED option from working), but note that some of the unfreed allocations might have been freed if the program had terminated successfully.

The mleak command scans through an mpatrol log file looking for lines beginning with ALLOC: and FREE: but ignores lines beginning with REALLOC:, so only the LOGALLOCS and LOGFREES options are necessary when running a program linked with the mpatrol library. Note that as a result of this, no attempt is made to account for resizing of memory allocations and so the total amount of memory used by the resulting unfreed allocations may not be entirely accurate.

This command will also read the unfreed allocations table produced by the SHOWUNFREED option in the log file if one is present. The entries in the table will be compared with the currently allocated entries and will be added if not already present. However, this behaviour can be disabled by supplying the --ignore option to the mleak command.

The mleak command can also be instructed to limit the number of lines of stack tracing information that it will display for each unfreed memory allocation. This is controlled by the --max-stack option which takes the maximum number of stack trace lines to display as an argument. If the number of lines is given as 0 (the default) then there will be no limit to the length of each stack trace.

The mleak command takes one optional argument which must be a valid mpatrol log filename but if it is omitted then it will use mpatrol.log as the name of the log file to use. The mleak command makes two passes over the log file so the file must be randomly-accessible. If the filename argument is given as - then the standard input file stream will be used as the log file. Note also that the mleak command supports the --help and --version options in common with the other mpatrol command line tools.

Note that mpatrol patch 5 (which can be downloaded from http://www.cbmamiga.demon.co.uk/mpatrol/patches/mpatrol_patch5.tar.gz) contains the source code to a FORTRAN 90 program called condenseleaklog which offers similar facilities to the mleak command, as well as the ability to trim and filter stack traces.

Node:The mpsym command, Next:, Previous:The mleak command, Up:Utilities

9.3 The mpsym command

Another utility program that is provided is called mpsym, which is used to parse a log file produced by the mpatrol library and uses a debugger to append symbol names and source level information to code addresses in stack tracebacks. This should be used if the USEDEBUG option is not supported on a particular platform or does not work properly with a specific program. It will replace all existing symbols and source level information associated with the stack tracebacks in the mpatrol log file and will display the resulting log file on the standard output file stream.

The first argument to mpsym must be the filename of the executable file that produced the mpatrol log file but if it is omitted then mpsym will use a.out as the name of the executable file to use. The mpsym command will read the symbol table and debugging sections from this file in order to map the code addresses that appear in the mpatrol log file into symbol names and source level information. If the executable file does not contain a symbol table then no symbol names will be available and if it does not contain the appropriate debugging sections then no source level information will be available either. Obviously, if the executable file is not the same as the one that created the mpatrol log file then the final output will be wrong.

The second argument to mpsym must be a valid mpatrol log filename but if it is omitted then mpsym will use mpatrol.log as the name of the log file to use, or progfile.log if it can't find that. The mpsym command makes two passes over the log file so the file must be randomly-accessable. Note also that the mpsym command supports the --help and --version options in common with the other mpatrol command line tools.

If the mpatrol library cannot determine the name of a symbol for display in the log file then it will mark the symbol as ???. If the debugger that is being used by mpsym also has problems determining the name of the symbol then you can pass the --skip option to mpsym to instruct it to skip any symbols marked as ??? in the log file.

The mpsym command currently uses gdb as the debugger with which to obtain the additional information about the code addresses in the mpatrol log file. It also makes use of several UNIX text processing commands, including perl if it is installed, in order to extract information from the debugger's output and from the log file. As a result, the mpsym command is only likely to work on UNIX platforms or on systems which have the necessary commands installed.

Node:The mpedit command, Next:, Previous:The mpsym command, Up:Utilities

9.4 The mpedit command

Yet another utility program that is provided is called mpedit, which is used to invoke a text editor on a given source file and optionally jump to a specific line number. It is used as a support command by the mpatrol library when the EDIT or LIST options are used but it can quite easily be used as a command in its own right if properly configured. Because it is a shell script it can be easily configured to support other editors, but this unfortunately limits it to UNIX platforms at the moment.

The first argument to mpedit must be the filename of the source file to be edited or listed. If the source file does not exist then the contents of the MPATROL_SOURCEPATH environment variable will be used to help locate the source file, even if the filename contained an absolute or relative path component. This environment variable should consist of a colon-separated list of directory names which may contain absolute paths or be relative to the current directory; the first directory in the list will be searched first. If the MPATROL_SOURCEPATH environment variable is not set then only the current directory will be searched. You can also use the --source-dir option to add directories to the search path used to locate the source file. Multiple --source-dir options may be given, and each will be prepended to the MPATROL_SOURCEPATH environment variable in the order given on the command line.

If the second argument specifying the line number is omitted then it is assumed to be 1. The text editor will attempt to jump to the specified line after opening the source file. The text editor that mpedit uses is controlled by setting the EDITOR environment variable. This can be set to the full pathname of the text editor to use or it can be set to the command that would normally be used to invoke the text editor, but it cannot also contain command line options. You can also use the --editor option to specify the text editor on the command line instead of using the value in the EDITOR environment variable.

The currently supported editors are vi, vim, elvis, emacs, xemacs, pico and nano, and if the EDITOR environment variable is not set then the default will be vi. Selecting an unsupported text editor will result in an error. However, you can edit the mpedit file to add support for your own favourite text editor as long as it supports a way to immediately jump to a specific line number when it is first started up. Note that the text editor must also open a new window to edit the source file so that it does not obscure any mpatrol diagnostic messages, and if it does not support this then a new terminal window must be opened for it to use.

If the --listing option is given on the command line then the mpedit command will display a context listing of the source file at the specified line number to the standard error output stream instead of invoking the text editor. The listing will be annotated with line numbers and will also show the contents of the five lines before and after the specified line if possible. Note also that the mpedit command supports the --help and --version options in common with the other mpatrol command line tools.

Node:The hexwords command, Previous:The mpedit command, Up:Utilities

9.5 The hexwords command

The final utility program that is provided is called hexwords, which is used to generate hexadecimal constants from a dictionary of known words. Such numerical constants can be used in source files for a variety of debugging problems, and problems with uninitialised variables are especially relevant since these special numbers will stand out if seen from within a debugger. For example, here are some common (and some not-so-common) 32-bit hexadecimal constants that can be used as debugging aids:

word hex constant
addedbad 0xaddedbad
allocate 0xa110ca7e
badlabel 0xbad1abe1
baseball 0xba5eba11
codebabe 0xc0debabe
codedbad 0xc0dedbad
deadbeef 0xdeadbeef
deadcode 0xdeadc0de
failsafe 0xfa115afe
feedface 0xfeedface
freedata 0xf4eeda7a
goodcode 0x600dc0de

As can be seen above, many decimal digits can be used to represent the letters that they most closely resemble, along with the hexadecimal digits A through F. This provides a much larger selection of words that can be matched, although the digits 3 and 8 cannot be used due to the lack of any similar-looking letters. The digits and their corresponding letters are given in the following table.

digit letter
0 O, o or Q
1 I, i or l
2 Z or z
3 -
4 q or R
5 S or s
6 G
7 J or T
8 -
9 g
A-F A-F
a-f a-f

The argument to hexwords must be a valid dictionary filename but if it is omitted then hexwords will use /usr/dict/words as the name of the dictionary file to use. If that cannot be found then hexwords will try /usr/lib/dict/words and /usr/share/dict/words. The dictionary file must be a plain text file that contains one word per line, otherwise few to no words will be matched.

The words that are matched from the dictionary file can be controlled by using the --match option, which sets the type of case-sensitivity to use. A setting of exact performs a case-sensitive comparison of all of the words in the dictionary file and the hexadecimal digits, whereas a setting of any does not. The lower and upper settings convert the words in the dictionary file to lower and upper case respectively before performing a case-sensitive comparison. The default case-sensitivity is exact.

The minimum and maximum number of letters that are matched are controlled by the --minimum and --maximum options. None of the hexadecimal numbers displayed will have any less or more digits that those specified with these options. The default minimum is 4 digits and the default maximum is 8 digits. Note also that the hexwords command supports the --help and --version options in common with the other mpatrol command line tools.

The hexwords command currently makes use of several UNIX text processing commands in order to extract the words and their hexadecimal equivalents. As a result, the hexwords command is only likely to work on UNIX platforms or on systems which have the necessary commands installed.

Node:Profiling, Next:, Previous:Utilities, Up:Top

10 Profiling

The mpatrol library has the capability to summarise the information it accumulated about the behaviour of dynamic memory allocations and deallocations over the lifetime of any program that it was linked and run with. This summary shows a rough profile of all memory allocations that were made, and is hence called profiling. There are several other different kinds of profiling provided with most compilation tools, but they generally profile function calls or line numbers in combination with the time it takes to execute them.

Memory allocation profiling is useful since it allows a programmer to see which functions directly allocate memory from the heap, with a view to optimising the memory usage or performance of a program. It also summarises any unfreed memory allocations that were present at the end of program execution, some of which could be as a result of memory leaks. In addition, a summary of the sizes and distribution of all memory allocations and deallocations is available.

A memory allocation call graph is also available for the programmer to be able to see the caller and callee relationships for all functions that allocated memory, either directly or indirectly. This graph is shown in a tabular form similar to that of gprof, but it can also be written to a graph specification file for later processing by dot. The dot and dotty commands are part of GraphViz, an excellent graph visualisation package that was developed at AT&T Bell Labs and is available for free download for UNIX and Windows platforms from http://www.research.att.com/sw/tools/graphviz/.

Only allocations and deallocations are recorded, with each reallocation being treated as a deallocation immediately followed by an allocation. For full memory allocation profiling support, call stack traversal must be supported in the mpatrol library and all of the program's symbols must have been successfully read by the mpatrol library before the program was run. The library will attempt to compensate if either of these requirements are not met, but the displayed tables may contain less meaningful information. Cycles that appear in the allocation call graph are due to recursion and are currently dealt with by only recording the memory allocations once along the call stack.

Memory allocation profiling is disabled by default, but can be enabled using the PROF option. This writes all of the profiling data to a file called mpatrol.out in the current directory at the end of program execution, but the name of this file can be changed using the PROFFILE option and the default directory in which to place these files can be changed by setting the PROFDIR environment variable. Sometimes it can also be desirable for the mpatrol library to write out the accumulated profiling information in the middle of program execution rather than just at the end, even if it is only partially complete, and this behaviour can be controlled with the AUTOSAVE option. This can be particularly useful when running the program from within a debugger, when it is necessary to analyse the profiling information at a certain point during program execution.

When profiling memory allocations, it is necessary to distinguish between small, medium, large and extra large memory allocations that were made by a function. The boundaries which distinguish between these allocation sizes can be controlled via the SMALLBOUND, MEDIUMBOUND and LARGEBOUND options, but they default to 32, 256 and 2048 bytes respectively, which should suffice for most circumstances.

The mprof command is a tool designed to read a profiling output file produced by the mpatrol library and display the profiling information that was obtained. The profiling information includes summaries of all of the memory allocations listed by size and the function that allocated them and a list of memory leaks with the call stack of the allocating function. It also includes a graph of all memory allocations listed in tabular form, and an optional graph specification file for later processing by the dot graph visualisation package.

The mprof command also attempts to calculate the endianness of the processor that produced the profiling output file and reads the file accordingly. This means that it is possible to use mprof on a SPARC machine to read a profiling output file that was produced on an Intel 80x86 machine, for example. However, this will only work if the processor that produced the profiling output file has the same word size as the processor that is running the mprof command. For example, reading a 64-bit profiling output file on a 32-bit machine will not work.

In addition, the profiling output file also contains the version number of the mpatrol library which produced it. If the major version number that is embedded in the profiling output file is newer that the version of mpatrol that mprof came with then mprof will refuse to read the file. You should download the latest version of mpatrol in that case. The reason for storing the version number is so that the format of the profiling output file can change between releases of mpatrol, but also allow mprof to cope with older versions.

Along with the options listed below, the mprof command takes one optional argument which must be a valid mpatrol profiling output filename but if it is omitted then it will use mpatrol.out as the name of the file to use. If the filename argument is given as - then the standard input file stream will be used as the profiling output file. Note also that the mprof command supports the --help and --version options in common with the other mpatrol command line tools.

--addresses
Specifies that different call sites from within the same function are to be differentiated and that the names of all functions should be displayed with their call site offset in bytes. This affects the direct allocation and memory leak tables, as well as the allocation call graph and the graph specification file.
--call-graph
Specifies that the allocation call graph should be displayed. This is not displayed by default as it can get very large for even a moderately sized profiling output file.
--counts
Specifies that certain tables should be sorted by the number of allocations or deallocations rather than the total number of bytes allocated or deallocated. This affects the direct allocation and memory leak tables, as well as the allocation call graph and the graph specification file.
--graph-file <file>
Specifies that the allocation call graph should also be written to a graph specification file for later visualisation with dot. If file is given as stdout or stderr then the corresponding file stream will be used as the target for the graph specification file.
--leaks
Specifies that memory leaks rather than memory allocations are to be written to the graph specification file. This option only affects the output from the --graph-file option.
--stack-depth <depth>
Specifies the maximum stack depth to use when calculating if one call site has the same call stack as another call site. This also specifies the maximum number of functions to display in a call stack. If depth is 0 then the call stack depth will be unlimited in size. The default call stack depth is 1. This affects the memory leak table.

We'll now look at an example of using the mpatrol library to profile the dynamic memory allocations in a program. However, remember that this example will only fully work on your machine if the mpatrol library supports call stack traversal and reading symbols from executable files on that platform. If that is not the case then only some of the features will be available.

The following example program performs some simple calculations and displays a list of numbers on its standard output file stream, but it serves to illustrate all of the different features of memory allocation profiling that mpatrol is capable of. The source for the program can be found in tests/profile/test1.c. 

23  /*
24   * Associates an integer value with its negative string equivalent in a
25   * structure, and then allocates 256 such pairs randomly, displays them
26   * then frees them.
27   */


30  #include <stdio.h>
31  #include <stdlib.h>
32  #include <string.h>


35  typedef struct pair
36  {
37      int value;
38      char *string;
39  }
40  pair;


43  pair *new_pair(int n)
44  {
45      static char s[16];
46      pair *p;

48      if ((p = (pair *) malloc(sizeof(pair))) == NULL)
49      {
50          fputs("Out of memory\n", stderr);
51          exit(EXIT_FAILURE);
52      }
53      p->value = n;
54      sprintf(s, "%d", -n);
55      if ((p->string = strdup(s)) == NULL)
56      {
57          fputs("Out of memory\n", stderr);
58          exit(EXIT_FAILURE);
59      }
60      return p;
61  }


64  int main(void)
65  {
66      pair *a[256];
67      int i, n;

69      for (i = 0; i < 256; i++)
70      {
71          n = (int) ((rand() * 256.0) / (RAND_MAX + 1.0)) - 128;
72          a[i] = new_pair(n);
73      }
74      for (i = 0; i < 256; i++)
75          printf("%3d: %4d -> \"%s\"\n", i, a[i]->value, a[i]->string);
76      for (i = 0; i < 256; i++)
77          free(a[i]);
78      return EXIT_SUCCESS;
79  }

After the above program has been compiled and linked with the mpatrol library, it should be run with the PROF option set in the MPATROL_OPTIONS environment variable. Note that mpatrol.h was not included as it is not necessary for profiling purposes.

If all went well, a list of numbers should be displayed on the screen and a file called mpatrol.out should have been produced in the current directory. This is a binary file containing the total amount of profiling information that the mpatrol library gathered while the program was running, but it contains concise numerical data rather than human-readable data. To make use of this file, the mprof command must be run. An excerpt from the output produced when running mprof with the --call-graph option is shown below33. 

                             ALLOCATION BINS

                          (number of bins: 1024)

                  allocated                          unfreed
       --------------------------------  --------------------------------
 size   count       %     bytes       %   count       %     bytes       %

    2       9    1.76        18    0.61       9    3.52        18    1.95
    3     105   20.51       315   10.61     105   41.02       315   34.16
    4     121   23.63       484   16.30     121   47.27       484   52.49
    5      21    4.10       105    3.54      21    8.20       105   11.39
    8     256   50.00      2048   68.96       0    0.00         0    0.00

total     512              2970             256               922

                            DIRECT ALLOCATIONS

                 (0 < s <= 32 < m <= 256 < l <= 2048 < x)

       allocated                       unfreed
--------------------------  --------------------------
bytes       %   s  m  l  x  bytes       %   s  m  l  x  count  function

 2970  100.00  %%             922  100.00  %%             512  new_pair

 2970          %%             922          %%             512  total

                               MEMORY LEAKS

                         (maximum stack depth: 1)

                unfreed                      allocated
----------------------------------------  ----------------
     %     bytes       %   count       %     bytes   count  function

100.00       922   31.04     256   50.00      2970     512  new_pair

             922   31.04     256   50.00      2970     512  total

                          ALLOCATION CALL GRAPH

                         (number of vertices: 3)

           allocated               unfreed
     ---------------------  ---------------------
index   bytes   s  m  l  x     bytes   s  m  l  x  function
-------------------------------------------------
[1]                                                _start [1]
         2970  %%                922  %%               main [3]
-------------------------------------------------
         2970  %%                922  %%               main [3]
[2]                                                new_pair [2]
-------------------------------------------------
         2970  %%                922  %%               _start [1]
[3]                                                main [3]
         2970  %%                922  %%               new_pair [2]

The first table shown is the allocation bin table which summarises the sizes of all objects that were dynamically allocated throughout the lifetime of the program. In this particular case, counts of all allocations and deallocations of sizes 1 to 1023 bytes were recorded by the mpatrol library in their own specific bin and this information was written to the profiling output file. Allocations and deallocations of sizes larger than or equal to 1024 bytes are counted as well and the total number of bytes that they represent are also recorded. This information can be extremely useful in understanding which sizes of data structures are allocated most during program execution, and where changes might be made to make more efficient use of the dynamically allocated memory.

As can be seen from the allocation bin table, 9 allocations of 2 bytes, 105 allocations of 3 bytes, 121 allocations of 4 bytes, 21 allocations of 5 bytes and 256 allocations of 8 bytes were made during the execution of the program. However, all of these memory allocations except the 8 byte allocations were still not freed by the time the program terminated, resulting in a total memory leak of 922 bytes.

The next table shown is the direct allocation table which lists all of the functions that allocated memory and how much memory they allocated. The s m l x columns represent small, medium, large and extra large memory allocations, which in this case are 0 bytes is less than a small allocation, which is less than or equal to 32 bytes, which is less than a medium allocation, which is less than or equal to 256 bytes, which is less than a large allocation, which is less than or equal to 2048 bytes, which is less than an extra large allocation. The numbers listed under these columns represent a percentage of the overall total and are listed as %% if the percentage is 100% or as . if the percentage is less than 1%. Percentages of 0% are not displayed.

The information displayed in the direct allocation table is useful for seeing exactly which functions in a program directly perform memory allocation, and can quickly highlight where optimisations can be made or where functions might be making unnecessary allocations. In the example, this table shows us that 2970 bytes were allocated over 512 calls by new_pair() and that 922 bytes were left unfreed at program termination. All of the allocations that were made by new_pair() were between 1 and 32 bytes in size.

We could now choose to sort the direct allocation table by the number of calls to allocate memory, rather than the number of bytes allocated, with the --counts option to mprof, but that is not relevant in this example. However, we know that there are two calls to allocate memory from new_pair(), so we can use the --addresses option to mprof to show all call sites within functions rather than just the total for each function. This option does not affect the allocation bin table so the new output from mprof with the --call-graph and --addresses options looks like: 

                            DIRECT ALLOCATIONS

                 (0 < s <= 32 < m <= 256 < l <= 2048 < x)

       allocated                       unfreed
--------------------------  --------------------------
bytes       %   s  m  l  x  bytes       %   s  m  l  x  count  function

 2048   68.96  69               0    0.00                 256  new_pair+20
  922   31.04  31             922  100.00  %%             256  new_pair+140

 2970          %%             922          %%             512  total

                               MEMORY LEAKS

                         (maximum stack depth: 1)

                unfreed                      allocated
----------------------------------------  ----------------
     %     bytes       %   count       %     bytes   count  function

100.00       922  100.00     256  100.00       922     256  new_pair+140

             922   31.04     256   50.00      2970     512  total

                          ALLOCATION CALL GRAPH

                         (number of vertices: 4)

           allocated               unfreed
     ---------------------  ---------------------
index   bytes   s  m  l  x     bytes   s  m  l  x  function
-------------------------------------------------
[1]                                                _start+100 [1]
         2970  %%                922  %%               main+120 [4]
-------------------------------------------------
         2048  %%                  0                   main+120 [4]
[2]                                                new_pair+20 [2]
-------------------------------------------------
          922  %%                922  %%               main+120 [4]
[3]                                                new_pair+140 [3]
-------------------------------------------------
         2970  %%                922  %%               _start+100 [1]
[4]                                                main+120 [4]
         2048  %%                  0                   new_pair+20 [2]
          922  %%                922  %%               new_pair+140 [3]

The names of the functions displayed in the above tables now have a byte offset appended to them to indicate at what position in the function a call to allocate memory occurred34. Now it is possible to see that the first call to allocate memory from within new_pair() has had all of its memory freed, but the second call (from strdup()) has had none of its memory freed.

This is also visible in the next table, which is the memory leak table and lists all of the functions that allocated memory but did not free all of their memory during the lifetime of the program. The default behaviour of mprof is to show only the function that directly allocated the memory in the memory leak table, but this can be changed with the --stack-depth option. This accepts an argument specifying the maximum number of functions to display in one call stack, with zero indicating that all functions in a call stack should be displayed. This can be useful for tracing down the functions that were indirectly responsible for the memory leak. The new memory leak table displayed by mprof with the --addresses and --stack-depth 0 options looks like: 

                               MEMORY LEAKS

                         (maximum stack depth: 0)

                unfreed                      allocated
----------------------------------------  ----------------
     %     bytes       %   count       %     bytes   count  function

100.00       922  100.00     256  100.00       922     256  new_pair+140
                                                            main+120
                                                            _start+100

             922   31.04     256   50.00      2970     512  total

Now that we know where the memory leak is coming from, we can fix it by freeing the string as well as the structure at line 77. A version of the above program that does not contain the memory leak can be found in tests/profile/test2.c.

The final table that is displayed is the allocation call graph, which shows the relationship between a particular function in the call graph, the functions that called it (parents), and the functions that it called (children). Every function that appears in the allocation call graph is displayed with a particular index that can be used to cross-reference it. The functions which called a particular function are displayed directly above it, while the functions that the function called are displayed directly below it. In the above example, _start() called main(), which then called new_pair() which allocated the memory.

The memory that has been allocated by a function (either directly, or indirectly by its children) for its parents is shown in the details for the parent functions, showing both a breakdown of the allocated memory and a breakdown of the unfreed memory. This also occurs for the child functions. If a function does not directly allocate memory then the total memory allocated for its parents will equal the total memory allocated by its children. However, if a parent or child function is part of a cycle in the call graph then a (*) will appear in the leftmost column of the call graph. In that case the total incoming memory may not necessarily equal the total outgoing memory for the main function.

In the example above when the --addresses option is used, it should be clear that new_pair()+20 allocates 2048 bytes for main(), while new_pair()+140 allocates 922 bytes for main(). The main() function itself allocates 2970 bytes for _start() overall via the new_pair() function.

It is also possible to view this information graphically if you have the GraphViz package mentioned above installed on your system. The --graph-file option can be used to write a dot graph specification file that can be processed by the dot or dotty commands that come with GraphViz. The resulting graphs will show the relationships between each function, its parents and its children, and will also show the number of bytes that were allocated along the edges of the call graph, but this can be changed to the number of calls if the --counts option is used35. A call graph showing unfreed memory instead of allocated memory can be generated by adding the --leaks option. The following graph illustrates the use of these options with the above example. It was generated using the --addresses and --graph-file options. images/test.jpg

As a final demonstration of mpatrol's profiling features we will attempt to profile a real application in order to see where the memory allocations come from. Since all of the following steps were performed on a Solaris machine, the --dynamic option of the mpatrol command was used to allow us to replace the system memory allocation routines with mpatrol's routines without requiring a relink. It also means that we can profile all of the child processes that were created by the application as well.

The application that we are going to profile is the GNU C compiler, gcc (version 2.95.2), which is quite a complicated and large program. The actual gcc command is really the compiler driver which invokes the C preprocessor followed by the compiler, the assembler, the prelinker and finally the linker (well, it does in this example). On Solaris, the gcc distribution uses the system assembler and linker which come with no symbol tables in their executable files so we will not be profiling them.

For the purpose of this demonstration we will only be looking at the graph files produced by the --graph-file option of the mprof command, but ordinarily you would want to look at the tables that mprof produces as well. All of the command line examples use the bash shell but in most cases these will work in other shells with a minimal amount of changes.

We will use tests/profile/test2.c as the source file to compile with gcc and we'll turn on optimisation in order to cause gcc to allocate a bit more memory than it would normally. Note that use is also made of the format string feature of the --log-file and --prof-file options so that it is clear which mpatrol log and profiling output files belong to which processes. 

bash$ mpatrol --dynamic --log-file=%p.log --prof-file=%p.out
              --prof gcc -O -o test2 test2.c
bash$ ls *.log *.out
as.log         cc1.out        cpp.log        gcc.out
as.out         collect2.log   cpp.out        ld.log
cc1.log        collect2.out   gcc.log        ld.out

As mentioned above, we're not interested in the mpatrol log and profiling output files for as and ld so we'll delete them. We can now use mprof to create graph specification files for each of the profiling output files produced. You can find these graph specification files and the profiling output files used to generate them in the extra directory in the mpatrol distribution. 

bash$ rm as.log as.out ld.log ld.out
bash$ ls *.out
cc1.out        collect2.out   cpp.out        gcc.out
bash$ for file in *.out
> do
>     mprof --graph-file=`basename $file .out`.dot $file
> done >/dev/null
bash$ ls *.dot
cc1.dot        collect2.dot   cpp.dot        gcc.dot

The graph specification files that have now been produced can be viewed and manipulated with the dotty command, or they can be converted to various image formats with the dot command. However, this presumes that you already have the GraphViz graph visualisation package installed. If you have then you can convert the graph specification files to GIF and postscript images using the following commands. If not, you can still view the graphs produced in the following figures. 

bash$ dot -Tgif -Gsize="6,3" -Gratio=fill -o gcc.gif gcc.dot
bash$ dot -Tgif -Gsize="6,3" -Gratio=fill -o cpp.gif cpp.dot
bash$ dot -Tgif -Gsize="7,4" -Gratio=fill -o cc1.gif cc1.dot
bash$ dot -Tgif -Gsize="4,3" -Gratio=fill -o collect2.gif collect2.dot
bash$ dot -Tps -Gsize="6,3" -Gratio=fill -o gcc.ps gcc.dot
bash$ dot -Tps -Gsize="6,3" -Gratio=fill -o cpp.ps cpp.dot
bash$ dot -Tps -Gsize="9,6" -Gratio=fill -Grotate=90 -o cc1.ps cc1.dot
bash$ dot -Tps -Gsize="4,3" -Gratio=fill -o collect2.ps collect2.dot
images/gcc.jpg

The figure above shows the allocation call graph for the gcc compiler driver. From the graph you can see that the vast majority of memory allocations appear to be required for reading the driver specification file, which details default options and platform-specific features. Almost all of the memory allocations go through the xmalloc() routine, which is an error-checking function built on top of malloc()36. A large amount of memory is also allocated by the obstack module, which provides a functionality similar to arenas for variable-sized data structures. You'll see extensive use of both of these types of routines throughout the following graphs. images/cpp.jpg

As would be expected, in the above allocation call graph for the cpp C preprocessor, the majority of memory allocations are used for macro processing, with a sizable chunk being allocated for reading include files. You may also notice the dotted lines that connect the rescan(), handle_directive(), do_include() and finclude() functions37. These show a cycle in the call graph where each of these functions may have been involved in one or more recursive calls. The labels for such dotted edges may not be entirely accurate since mprof will only count allocated memory once for each recursive call chain.

The following figure shows the allocation call graph for the cc1 compiler itself. As you would expect, it's a bit of a beast compared to the previous two graphs, and looks very hard to follow. However, if you look closer you will notice that the various groups of functions that comprise the compiler stand out due to their close association with one another. For example, you might notice that the functions between cse_insn() and get_cse_reg_info() form a group that allocates 9140 bytes overall. You can also see the parser module at the top left of the graph, initiated with yyparse(), and the code generator module in the rest of the graph, initiated with rest_of_compilation(). images/cc1.jpg images/collect2.jpg

The allocation call graph for the prelinker, collect2, is a lot simpler than the previous graphs. There are no cycles in the graph and most of the allocations are concerned with maintaining hash tables. Once again, xmalloc() and _obstack_begin() are the two main sources of memory allocation.

As can be seen, a lot of information about the memory allocation behaviour of a program can be obtained by creating a visual image of the allocation call graph. In addition, different graphs can be produced to show call counts instead of allocated bytes (via the --counts option), and graphs of unfreed memory can be produced to detect where memory leaks come from (via the --leaks option).

Although mprof does not currently offer this facility, a small tool called profdiff which reports differences between two mpatrol profiling output files can be downloaded from http://www.cbmamiga.demon.co.uk/mpatrol/patches/mpatrol_patch3.tar.gz.

Much of the functionality of this implementation of memory allocation profiling is based upon mprof by Benjamin Zorn and Paul Hilfinger, which was written as a research project and ran on MIPS, SPARC and VAX machines. However, the profiling output files are incompatible, the tables displayed have a different format, and the way they are implemented is entirely different.

Node:Tracing, Next:, Previous:Profiling, Up:Top

11 Tracing

In addition to profiling, the mpatrol library also has the capability to concisely trace the details of every dynamic memory allocation, reallocation and deallocation over the lifetime of any program that it was linked and run with. This information can then be used to calculate trends in a program's memory allocation behaviour and provide details on the lifetimes of memory allocations. In contrast to profiling, it can also be used to display a program's memory allocation behaviour in real-time, along with some useful information that can be displayed in graphical or tabular form.

Only allocations, reallocations and deallocations are recorded. The intention of tracing is to gather concise details about each memory allocation event rather than complete information about some or all memory allocations. As a result, the mpatrol log files and profiling output files contain more detailed information about individual memory allocations, whereas the tracing output files contain a broader view of allocation behaviour throughout the entire program.

Memory allocation tracing is disabled by default, but can be enabled using the TRACE option. This writes all of the tracing data to a file called mpatrol.trace in the current directory at the end of program execution, but the name of this file can be changed using the TRACEFILE option and the default directory in which to place these files can be changed by setting the TRACEDIR environment variable.

The mptrace command is a tool designed to read a tracing output file produced by the mpatrol library and display the tracing information that was obtained. The tracing information is a concise encoded trace of all of the memory allocation events that occurred during a program's execution, and can be decoded into tabular or graphical form, along with any relevant statistics that can be calculated.

The mptrace command also attempts to calculate the endianness of the processor that produced the tracing output file and reads the file accordingly. This means that it is possible to use mptrace on a SPARC machine to read a tracing output file that was produced on an Intel 80x86 machine, for example. However, this will only work if the processor that produced the tracing output file has the same word size as the processor that is running the mptrace command. For example, reading a 64-bit tracing output file on a 32-bit machine will not work.

In addition, the tracing output file also contains the version number of the mpatrol library which produced it. If the major version number that is embedded in the tracing output file is newer that the version of mpatrol that mptrace came with then mptrace will refuse to read the file. You should download the latest version of mpatrol in that case. The reason for storing the version number is so that the format of the tracing output file can change between releases of mpatrol, but also allow mptrace to cope with older versions.

Along with the usual --help and --version options, the mptrace command accepts several other options and takes one optional argument which must be a valid mpatrol tracing output filename but if it is omitted then it will use mpatrol.trace as the name of the file to use. If the filename argument is given as - then the standard input file stream will be used as the tracing output file.

Normally, the mptrace command will simply read in the tracing output file and display any statistics it has gathered. However, it can also be instructed to display a tracing table which displays useful details for every event in the tracing output file. The tracing table can be displayed with the --verbose option. If the mpatrol library was able to obtain source-level information for one or more memory events then this can be displayed in the tracing table by adding the --source option.

A C source file containing a trace-driven memory allocation simulation program can be written with the --sim-file option. This program will have the identical memory allocation behaviour to the program which produced the original trace file. This option can be useful to use if you wish to determine which malloc library is most suitable to use for a specific application.

A trace file in Heap Allocation Trace Format (HATF) can also be written out by the mptrace command by using the --hatf-file option. It takes the name of the HATF trace file to be written as an argument and writes out the HATF version of the mpatrol tracing output file given as input when it is being processed. The HATF file format is an attempt to standardise trace file formats for memory allocation tracing, and is being developed by Benjamin Zorn, Richard Jones and Trishul Chilimbi. There is a HATF DTD located in the extra directory in the mpatrol distribution.

The mptrace command will normally be built with GUI38 support on UNIX platforms that are running X Windows. This means that a graphical memory map display of the heap will be shown in a window every time mptrace is run with the --gui option. This display is updated every time a new event is read from the tracing output file and by default uses the colour red for internal heap memory (used by the mpatrol library), blue for unallocated heap memory, black for allocated memory and white for free memory. Options exist to change this colour scheme, as well as the dimensions of the drawing area and the window.

By default, it is assumed that the start address of the first event that appears in the tracing output file is the base address of the memory map displayed in the window. If the heap grows downwards then this assumption will be incorrect (since nothing will be displayed) and so the --base option must be used to specify a reasonable lower bound for the final memory map. In addition, the visible address space displayed in the memory map is fixed to a certain size (4 megabytes by default), but this can be changed with the --space option. A small delay can also be added after drawing each memory allocation event through the use of the --delay option.

The following options are specific to the GUI version of mptrace and are read by the X command line parser rather than directly by mptrace. As a result they are parsed according to X toolkit rules and do not appear in the quick-reference option summary produced by the --help option. The application class for setting mptrace X resources is called MPTrace.

--alloc <colour>
Specifies the colour to use for displaying allocated memory. The default colour is black.
--base <address>
Specifies the base address of the visible address space displayed in the memory map. The default address is calculated at run-time from the start address of the first memory allocation event in the tracing output file.
--delay <length>
Specifies that a small delay of a certain length should be added after drawing each memory allocation event. The delay does not correspond to a specific unit of time, but experimentation with the length should yield satisfactory results. The default delay is 0.
--free <colour>
Specifies the colour to use for displaying free memory. The default colour is white.
--height <size>
Specifies the height (in pixels) of the drawing area. The default height is 512.
--internal <colour>
Specifies the colour to use for displaying internal heap memory. The default colour is red.
--space <size>
Specifies the size (in megabytes) of the visible address space displayed in the memory map. The default size is 4.
--unalloc <colour>
Specifies the colour to use for displaying unallocated heap memory. The default colour is blue.
--view-height <size>
Specifies the height (in pixels) of the window. The default height is 256.
--view-width <size>
Specifies the width (in pixels) of the window. The default width is 256.
--width <size>
Specifies the width (in pixels) of the drawing area. The default width is 512.

We'll now look at an example of using the mpatrol library to trace the dynamic memory allocations in a program. As with the previous chapter we will attempt to trace a real application in order to examine its memory allocation behaviour. Since all of the following steps were performed on a Solaris machine, the --dynamic option of the mpatrol command was used to allow us to replace the system memory allocation routines with mpatrol's routines without requiring a relink. It also means that we can trace all of the child processes that were created by the application as well.

The application that we are going to trace is the GNU C compiler, as before, and we will discard the tracing information generated for the assembler and linker. All of the command line examples use the bash shell but in most cases these will work in other shells with a minimal amount of changes.

We will use tests/profile/test2.c as the source file to compile with gcc and we'll turn on optimisation in order to cause gcc to allocate a bit more memory than it would normally. Note that use is also made of the format string feature of the --log-file and --trace-file options so that it is clear which mpatrol log and tracing output files belong to which processes. 

bash$ mpatrol --dynamic --log-file=%p.log --trace-file=%p.trace
              --trace gcc -O -o test2 test2.c
bash$ ls *.log *.trace
as.log         cc1.trace      cpp.log        gcc.trace
as.trace       collect2.log   cpp.trace      ld.log
cc1.log        collect2.trace gcc.log        ld.trace

As mentioned above, we're not interested in the mpatrol log and tracing output files for as and ld so we'll delete them. We can now use mptrace to decode each of the tracing output files produced and write their contents in tabular form to the standard output file stream, which can be redirected to a file for later viewing. You can find these tracing output files in the extra directory in the mpatrol distribution.

Note that both the tracing files mentioned above and the examples below treat reallocations as a deallocation followed by an allocation. This was the behaviour in older versions of the mpatrol library and I haven't bothered to update the files. However, it shouldn't affect the final outcome in any way. In addition, as the mpatrol.h header file was not included by any of the source files that comprise the compiler and its toolset, there was no source-level information for memory events. If there was, the --source option could have been used to display it. 

bash$ rm as.log as.trace ld.log ld.trace
bash$ ls *.trace
cc1.trace      collect2.trace cpp.trace      gcc.trace
bash$ for file in *.trace
> do
>     mptrace --verbose $file >`basename $file .trace`.res
> done
bash$ ls *.res
cc1.res        collect2.res   cpp.res        gcc.res

For the purposes of this example we will only be looking at the tracing results for the cc1 compiler which are now decoded in the file cc1.res. If you examine this file you will see something similar to the following. Note that the ... marks text that has been removed. 

 event  type     index  allocation      size    life   count     bytes
------  ------  ------  ----------  --------  ------  ------  --------
        internal        0x0024E000     32768
        internal        0x00256000     32768
        internal        0x0025E000     32768
        reserve         0x00266000      8192
        internal        0x00268000     32768
        internal        0x00270000     32768
        internal        0x00278000     32768
        internal        0x00280000     32768
        internal        0x00288000     32768
        internal        0x00290000     32768
...
        reserve         0x00308000     16384
     1  alloc       19  0x00266568      4072               1      4072
     2  alloc       21  0x0030A008      4072               2      8144
     3  alloc       22  0x0030AFF0      4072               3     12216
        reserve         0x0030C000      8192
     4  alloc       23  0x0030BFD8      4072               4     16288
     5  alloc       24  0x0030CFC0      4072               5     20360
        reserve         0x0030E000      8192
     6  alloc       25  0x0030DFA8      4072               6     24432
     7  alloc       26  0x00267550        42               7     24474
...
  1712  free       650  0x00373FF0      4072     827     398    321191
  1713  free       649  0x00376FA8      4072     829     397    317119
  1714  alloc     1074  0x00376FA8      4072             398    321191
  1715  free       233  0x0031ED18      8200    1498     397    312991
  1716  free       234  0x00320D20      8200    1498     396    304791
  1717  free       299  0x00355CC8       620    1426     395    304171
  1718  free       655  0x00353A28      1016     823     394    303155
  1719  free       303  0x0035E000      5096    1424     393    298059
  1720  free       653  0x00354E60       152     827     392    297907
  1721  free       654  0x00354EF8       152     827     391    297755

memory allocation tracing statistics
------------------------------------
allocated: 1056 (540776 bytes)
freed:     665 (243021 bytes)
unfreed:   391 (297755 bytes)
peak:      489 (375169 bytes)
reserved:  48 (409600 bytes)
internal:  27 (884736 bytes)
total:     75 (1294336 bytes)

smallest size: 3 bytes
largest size:  8200 bytes
average size:  512 bytes

There are eight different columns of data displayed by the mptrace command when it decodes the tracing output file and displays it in tabular format with the --verbose option. Here is an explanation for each of them.

event
This contains the event number (or time line) for each memory allocation, reallocation or deallocation (heap reservations are not considered events for this purpose). Each memory allocation, reallocation or deallocation increases the current event number, and this information is used to calculate the lifetime of a heap allocation.
type
This contains the event type for each entry in the tracing output file. Memory allocations, reallocations and deallocations are represented by alloc, realloc and free respectively. Normal heap reservations (that will be used for memory allocations) are represented by reserve, while internal heap reservations (for use by the mpatrol library itself) are represented by internal.
index
This contains the allocation index that is used by the mpatrol library to keep track of each unique memory allocation, and corresponds directly to any memory allocations listed in the log file. In older tracing output files, memory allocation events that reuse allocation indices represent a reallocation of the original allocation.
allocation
This contains the start address of the memory allocation.
size
This contains the size (in bytes) of the memory allocation.
life
This contains the lifetime of a memory allocation and is displayed when it is is freed. It is simply the difference between the current event number and the event number at which the original allocation took place, but is useful for working out how long a memory allocation is valid throughout a program's execution. If a memory allocation is reallocated, its lifetime will be calculated from the original time of allocation, not the point at which it was reallocated.
count
This contains a running total of the number of memory allocations currently in use. The total is calculated after processing the current event.
bytes
This contains a running total of the memory used by the current memory allocations. The total is calculated after processing the current event.

The first few entries in the table show that the mpatrol library started by allocating memory from the heap for its own purposes before reserving 8192 bytes for the memory allocations made by the object file access library for reading the symbols from the executable file and shared libraries39. Most of the further internal heap reservation events are due to the mpatrol library having to store details for all of the relevant symbols that it could read at program startup. The more symbols that there are, the more memory that must be used to store them. Note that the heap reservation events are not really relevant to the analysis of the program's memory allocations but they are used when displaying the heap graphically.

The first few memory allocation events in the table show that several memory allocations of 4072 bytes are being made along with several more heap reservations that are needed to store them. The last events in the table are mainly all deallocation events of allocations that were made quite early on in the program. The lifetime information for these events shows that some of these allocations were made very near the beginning of the program, while the others were made near the middle. None of them were very big and so would not be occupying much memory.

The statistics that were gathered from the tracing output file are displayed after the tracing table. The first group of entries summarise the heap memory that was used, with the allocated, freed and unfreed fields showing the total number of memory allocations that were made, the total number of memory allocations that were freed, and the total number of unfreed memory allocations respectively. The peak field shows the highest number of memory allocations (and total number of bytes) that were in use at any one time. The reserved and internal fields show the total number of pages reserved from the system heap for user allocations and internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written out to the tracing output file.  If stdout is not
# suitable then stderr could be used instead if you redirect it.

bash$ mpatrol --trace-file=stdout --trace ./testprog | mptrace --verbose -

# This method creates a named pipe called myfifo (but it could be
# called anything) and runs the program being traced and the mptrace
# command separately (perhaps in two separate windows).  If the
# mkfifo command is not available on your system then try mknod.

bash$ mkfifo myfifo
bash$ mpatrol --trace-file=myfifo --trace ./testprog &
bash$ mptrace --verbose myfifo

The idea for graphically displaying a memory map of the heap comes from the xmem tool supplied with the University of Toronto Computer Systems Research Institute malloc library, written by Mark Moraes. However, the documentation for that tool remarks that it was written as a quick and dirty hack. The mptrace command is hopefully more stable and contains a lot more functionality.

The mpatrol library can also generate trace files in a format that is compatible with the GNU mtrace() option. The code to do this is built on top of the mpatrol library and is in tools/mtrace.c and tools/mtrace.h. Such trace files can then be processed by the GNU mtrace command. The tools/mgauge.c and tools/mgauge.h files in the same directory can be used to implement an allocated memory gauge which updates in real-time in a terminal window. This can be used as an alternative to the window used by the mptrace command's --gui option for a simpler display.

Node:Heap corruption, Next:, Previous:Tracing, Up:Top

12 Heap corruption

There can be many causes of heap corruption in a program and there can be many forms in which it can appear. This chapter attempts to describe the most appropriate ways to narrow down and remove the causes of the most common forms of heap corruption. Note that errors such as freeing an allocated block twice are not considered in this chapter even though they would result in heap corruption in a normal malloc library -- the mpatrol library catches these special cases so you know exactly where they occur.

The three forms of errors we are going to look at are heap corruption in free memory blocks, freed memory blocks and overflow buffers. As you will soon see, the same piece of faulty code can produce any one of these errors depending on which mpatrol library options you use. The following discussion assumes that you have run your program with the mpatrol library and you get an ALLOVF, FRDCOR, FRDOVF or FRECOR error in the mpatrol log file when your program terminates. It also assumes that you haven't set the MPATROL_OPTIONS environment variable yet.

By default, the only times the mpatrol library will check the heap for memory corruption are when it terminates or when __mp_check() is called (but the latter won't be happening since you won't have modified your program yet). This isn't good enough for errors such as these so we need to instruct it to make checks whenever an mpatrol library function is called. The CHECK option controls when such automated checks occur, and this can normally be set to CHECK=- to check the heap whenever a call to an mpatrol library function is made.

However, in programs which take a long time to execute, or programs which make a large number of memory allocations, this can slow the program down quite a bit so you might want to try the optional /freq argument to the CHECK option. This simply instructs the mpatrol library to make the checks every freq calls to the mpatrol library functions rather than every call. For example, CHECK=/10 will make the checks every 10 calls, which will reduce the slowdown in the program but will still help narrow down where the heap corruption is occurring.

We'll use the following program as a running example for the discussions below, although you'll probably be following them using your program instead of this one. It contains a small bug that doesn't normally show up when using the system C library but causes a FRECOR error when linked with mpatrol. 

 1  /*
 2   * A program which causes heap corruption.
 3   */


 6  #include <stdio.h>
 7  #include "mpatrol.h"


10  int main(void)
11  {
12      char *p[128];
13      size_t i;

15      for (i = 0; i < 128; i++)
16      {
17          if ((p[i] = (char *) malloc(9)) == NULL)
18          {
19              fputs("out of memory\n", stderr);
20              exit(EXIT_FAILURE);
21          }
22          sprintf(p[i], "test%lu", i * 100);
23          puts(p[i]);
24          free(p[i]);
25      }
26      return EXIT_SUCCESS;
27  }

We get the following error in the mpatrol log file when we run with the above example linked to the mpatrol library. The error occurs when the program returns from main() since that is when the mpatrol library is terminati internal allocations respectively, and the total field shows the total number of pages that were used from the system heap.

The smallest size and largest size fields indicate the sizes of the smallest memory allocation and the largest memory allocation respectively. The average size field shows the mean number of bytes that was allocated between each of the memory allocations.

If you were running a GUI version of mptrace, information about all of these events can be displayed in graphical form inside a window if the --gui option is used. The following screenshot shows the mptrace display window when it is run with the --gui option and cc1.trace as input. It was generated using the --space 2 option. images/trace.jpg

Areas coloured blue indicate heap memory that has not yet been used by the mpatrol library (i.e. it has not currently been allocated from the system, or is currently being used by a part of the program that is not being tracked by the mpatrol library). Areas coloured red indicated heap memory that is being used internally by the mpatrol library. In this example, the reason that there is so much internal memory being used is that there are a large number of symbols that were read from the executable file and shared libraries. The narrow band of black and white lines at the top of the memory map represents the memory that was used by the object file access library when it was reading the symbols.

The large black bands in the middle of the memory map indicate memory that was still allocated at program termination. While this is a substantial amount compared to the amount of free memory, it does not necessary indicate memory leaks as the memory could be being used right up until the end of the program, and is implicitly freed at program termination anyway.

Unlike memory allocation profiling which summarises all of the accumulated data, it is possible to trace memory allocation events in real-time as the program runs. This can currently be done on UNIX platforms by piping the tracing output file from the program being run to the mptrace command, which can be achieved in several ways depending on the UNIX system that you are using. Both of the following methods are equivalent, where testprog is the name of the program that is being traced (and has previously been linked with the mpatrol library). 

# This method specifies the standard output file stream as the
# destination for the tracing output file and then runs both
# commands in a shell command pipe.  This has a disadvantage in
# that testprog must not write anything to stdout since that would
# be written o