<?xml version='1.0'?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-
open.org/docbook/xml/4.5/docbookx.dtd" [
]>

<chapter id="userspace-probing">
	<title>User-Space Probing</title>

<para>
SystemTap initially focused on kernel-space probing.  However,
there are many instances where user-space probing can help diagnose
a problem.  SystemTap 0.6 added support to allow probing user-space processes.
SystemTap includes support for probing the entry into and return from a
function in user-space processes, probing predefined markers in
user-space code, and monitoring user-process events.
</para>

<para>
The SystemTap user-space probing requires the utrace kernel
extensions which provide an API for tracking various user-space events.
More details about the utrace infrastructure are available at 
<ulink url="http://sourceware.org/systemtap/wiki/utrace" />.
The following command determines whether the currently running Linux kernel
provides the needed utrace support:
</para>

<screen>grep CONFIG_UTRACE /boot/config-`uname -r`</screen>

<para>
If the Linux kernel support user-space probing, the following output is printed:
</para>

<screen>CONFIG_UTRACE=y</screen>

<para>
The SystemTap user-space probing also needs the uprobes kernel
module. If the uprobes kernel module is not available, you will
see an error message like the following when attempting to run a script that requires
the uprobes kernel module:
</para>

<screen>SystemTap's version of uprobes is out of date.
As root, or a member of the 'root' group, run
"make -C /usr/share/systemtap/runtime/uprobes".
Pass 4: compilation failed.  Try again with another '--vp 0001' option.</screen>

<para>
If this occurs, you need to generate a uprobes.ko module for the kernel as directed.
</para>

<!-- FIXME How does uprobes work for cross compilation and
 using a compile server? -->


<section id="uevents">
	<title>User-Space Events</title>

<indexterm>
<primary>Events</primary>
<secondary>user-space</secondary>
</indexterm>

<para>
All user-space event probes begin with <emphasis>process</emphasis>.
The process events can be limited to a specific running process by specifying the process ID.
The process events can also be limited to monitoring a particular executable
by specifying the path to executable (PATH).
SystemTap makes use of the <command>PATH</command> environment variable,
so both the name used on the command-line to start the executable and the absolute path
to the executable can be used.
Several of user-space probe events limit their scope to a
particular executable name (PATH) because
SystemTap must use debug information to statically analyzed where to
places the probes, but for many user-space probes events the process ID
and executable name are optional. 
Any <command>process</command> event in the list below that include process ID or
the path to the executable must include those arguments.
The process ID and path to the executable are optional for the
<command>process</command> events that do not list them:
</para>

<variablelist>

<varlistentry>
<term>process("<replaceable>PATH</replaceable>").function("<replaceable>function</replaceable>")</term>
<listitem>
<para>
The entry to the user-space function <replaceable>function</replaceable> for the executable <replaceable>PATH</replaceable>.
This event is the user-space analogue of the
<command>kernel.function("<replaceable>function</replaceable>")</command> event.
It allows wildcards for the function <replaceable>function</replaceable>
and <command>.return</command> suffix.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term>process("<replaceable>PATH</replaceable>").statement("<replaceable>statement</replaceable>")</term>
<listitem>
<para>
The earliest instruction in the code for <replaceable>statement</replaceable>. This is the user-space analogue of 
<command>kernel.statement("<replaceable>statement</replaceable>")</command>.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term>process("<replaceable>PATH</replaceable>").mark("<replaceable>marker</replaceable>")</term>
<listitem>
<para>The static probe point <replaceable>marker</replaceable> defined in
<replaceable>PATH</replaceable>.
Wildcards can be used for <replaceable>marker</replaceable> to specify
mutiple marks with a single probe.
The static probe points may also have numbered arguments ($1, $2, etc.)
available to the probe.
A variety of user-space packages such as Java include these static
probe points.
Most packages that provide static probe points also
provide aliases for the raw user-space mark events.
Below is one such alias for the x86_64 Java hotspot JVM:
 </para>
<screen>probe hotspot.gc_begin =
  process("/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server/libjvm.so").mark("gc__begin")</screen>
</listitem>
</varlistentry>

<varlistentry>
<term>process.begin</term>
<listitem>
<para>User-space process is created.
This can be limited to a a particular process ID or a full path to the
executable.</para>
</listitem>
</varlistentry>

<varlistentry>
<term>process.thread.begin</term>
<listitem>
<para>User-space thread is created.
This can be limited to a a particular process ID or a full path to the
executable.</para>
</listitem>
</varlistentry>

<varlistentry>
<term>process.end</term>
<listitem>
<para>User-space process died.
This can be limited to a a particular process ID or a full path to the
executable.</para>
</listitem>
</varlistentry>

<varlistentry>
<term>process.thread.end</term>
<listitem>
<para>User-space thread is destroyed.
This can be limited to a a particular process ID or a full path to the
executable.</para>
</listitem>
</varlistentry>

<varlistentry>
<term>process.syscall</term>
<listitem>
<para>User-space process makes a system call.
The system call number is available via $syscall context variable, and 
the fist six arguments are available via $arg1 through $arg6.
The ".return" suffix will place the probe at the return from the system call.
For the "syscall.return" the return value is available through the
$return context variable.
This can be limited to a a particular process ID or a full path to the
executable.</para>
</listitem>
</varlistentry>

</variablelist>


</section>

<section id="utargetvariable">
	<title>Accessing User-Space Target Variables</title>

<indexterm>
<primary>target variables</primary>
<secondary>user-space</secondary>
</indexterm>

<para>
User-space target variables can be accessed in the same manner as described in
<xref linkend="targetvariables"/>.
However, in Linux there are separate address spaces for the user and
kernel code.
When using "->" operator SystemTap will access the appropriate address space.
For pointers to base types such as integers and strings
there are a number of functions listed below to access user-space data.
The first argument for each functions is the pointer to the data item.
</para>

<variablelist>

<varlistentry>
<term>user_char(<replaceable>address</replaceable>)</term>
<listitem>
<para>
Obtain the character at <replaceable>address</replaceable> for the current
user process.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term>user_short(<replaceable>address</replaceable>)</term>
<listitem>
<para>
Obtain the short at <replaceable>address</replaceable> for the current
user process.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term>user_int(<replaceable>address</replaceable>)</term>
<listitem>
<para>
Obtain the int at <replaceable>address</replaceable> for the current
user process.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term>user_long(<replaceable>address</replaceable>)</term>
<listitem>
<para>
Obtain the long at <replaceable>address</replaceable> for the current
user process.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term>user_string(<replaceable>address</replaceable>)</term>
<listitem>
<para>
Obtain the string at <replaceable>address</replaceable> for the current
user process.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term>user_string_n(<replaceable>address</replaceable>, <replaceable>n</replaceable>)</term>
<listitem>
<para>
Obtain the string at <replaceable>address</replaceable> for the current
user process and limits the string to <replaceable>n</replaceable> bytes.
</para>
</listitem>
</varlistentry>

</variablelist>

</section>

<section id="ustack">
	<title>User-Space Stack Backtraces</title>

<indexterm>
<primary>Stack backtrace</primary>
<secondary>user-space</secondary>
</indexterm>

<!--
FIXME why is this useful?
FIXME describe how it works
FIXME cautions about debuginfo
FIXME when was -d -ldd options introduced?
FIXME description of how to use the stack backtracing
-->

<para>
      The probe point (<command>pp</command>) function indicates which
      particular event triggered the the SystemTap event handler.
      A probe on the entry into a function would list the function name.
      However, in many cases the same probe point event may be triggered by
      many different modules in the program; this is particularly true for
      functions in shared libraries. A SystemTap backtrace of the user-space
      stack can provide additional context on how the probe point event is
      triggered.
</para>

<para>
      The user-space stack backtrace generation is complicated 
      by the compiler producing code optimized to eliminate stack frame
      pointers.
      However, the compiler also includes information in the debug information
      section to allow debugging tools to produce stack backtraces.
      SystemTap user-space stack backtrace mechanism makes use of that
      debug information to walk the stack to generate stack traces for 32-bit
      and 64-bit x86 processors; other processor architectures do not yet
      support the use of debug information to unwind the user-space stack.
      You will need to use the
      <command>-d <replaceable>executable</replaceable></command>
      for the application executable and <command>-ldd</command>
      for shared libraries to ensure that the needed debug information
      is used to produce the user-space stack backtraces.
</para>

<para>
      If you want to see how the function <command>xmalloc</command> function is
      being called by the command <command>ls</command>, you could use the
      user-space backtrack functions to provide that information.  With the
      debuginfo for the <command>ls</command> command installed the following
      SystemTap command will provide a backtrace each time the
      <command>xmalloc</command> function is called:
</para>

<screen>stap -d /bin/ls --ldd \
-e 'probe process("ls").function("xmalloc") {print_ustack(ubacktrace())}' \
-c "ls /"</screen>

<para>
When the SystemTap script runs will have output similar to the following:
</para>

<screen>
bin	dev   lib	  media  net	     proc	sbin	 sys  var
boot	etc   lib64	  misc	 op_session  profilerc	selinux  tmp
cgroup	home  lost+found  mnt	 opt	     root	srv	 usr
 0x4116c0 : xmalloc+0x0/0x20 [/bin/ls]
 0x4116fc : xmemdup+0x1c/0x40 [/bin/ls]
 0x40e68b : clone_quoting_options+0x3b/0x50 [/bin/ls]
 0x4087e4 : main+0x3b4/0x1900 [/bin/ls]
 0x3fa441ec5d : __libc_start_main+0xfd/0x1d0 [/lib64/libc-2.12.so]
 0x402799 : _start+0x29/0x2c [/bin/ls]
 0x4116c0 : xmalloc+0x0/0x20 [/bin/ls]
 0x4116fc : xmemdup+0x1c/0x40 [/bin/ls]
 0x40e68b : clone_quoting_options+0x3b/0x50 [/bin/ls]
 0x40884a : main+0x41a/0x1900 [/bin/ls]
 0x3fa441ec5d : __libc_start_main+0xfd/0x1d0 [/lib64/libc-2.12.so]
 ...
</screen>

<para>
      For more details on the functions available for user-space stack
      backtraces look at the <command>ucontext-symbols.stp</command> and
      <command>ucontext-unwind.stp</command> tapsets. The descriptions
      of the functions in those tapsets can also be found in the
      <citetitle>SystemTap Tapset Reference Manual</citetitle>.
</para>


</section>

</chapter>
