• SystemtapOnDebian

In this document I will try to provide a step-by-step guide on how to install systemtap on a debian system (sid)

Get a working compiler

Some versions of GCC are known to miscompile debug information. Without proper debug information, Kprobes and SystemTap will be unable to work properly. Besides that, SystemTap requires the same compiler version used to build the kernel. So, save yourself some time and just go ahead and grab a good GCC compiler and recompile your kernel.

GCC 4.1.2 is a good compiler. On the other hand, the default GCC version supplied with Ubuntu 6.06 and Debian sarge, GCC 4.0.x, is an example of a bad compiler.

For more information on this issue see debian bug #378628 and debian bug #378621.

Get SystemTap

There are two ways of obtaining SystemTap.

Get SystemTap package

First of all, we will need SystemTap's package. As of 2006-06-29 it is available in testing and in unstable. A simple "apt-get install systemtap" should do the trick.

SystemTap is a fast evolving piece of software so rather than sticking to testing's version I suggest using unstable's. If you are not comfortable running unstable or pinning SystemTap from unstable ¹, you can always compile its package from source. Just make sure you have a deb-src entry pointing to an unstable repository in your sources.list and follow the procedure below:

apt-get build-dep systemtap
apt-get --compile source systemtap
dpkg -i systemtap*deb

Build a package with fresh files from CVS

If you are feeling adventurous, you can always build a package with fresh files from the CVS.

• This is not the best way to obtain the newest and best from SystemTap and I cannot guarantee this will produce a "sane" package either. YMMV. Anyway, you've been warned.

  In the lines below, replace version and timestamp values to match those present in the original Debian source package.

• Grab the systemtap source package to use as skeleton:

  • apt-get build-dep systemtap
    apt-get source systemtap
    cd systemtap-0.0.20060610

• Move the old systemtap tarball out of the way...

  • mv systemtap-20060610.tar.bz2 systemtap-20060610.tar.bz2.old

• Grab fresh files from the CVS and make a new tarball for systemtap

  • cvs -d :pserver:anoncvs@sources.redhat.com:/cvs/systemtap login
    # enter "anoncvs" as the password
    cvs -d :pserver:anoncvs@sources.redhat.com:/cvs/systemtap co systemtap
    tar jcvf systemtap-20060610.tar.bz2 --exclude=CVS src/

• Update the release number in debian/changelog.

  • Just edit the first line of this file. For instance,

    systemtap (0.0.20060610-1) unstable; urgency=low

    became

    systemtap (0.0.20060610-1cvs20060620) unstable; urgency=low

• Build the package

  • fakeroot dpkg-buildpackage -b

If dpkg-buildpackage fails while trying to apply debian/patches/03-systemtap-manpages, just remove this file

Get recent SystemTap from git tree

Generally recent systemtap is stable, so it is safe to use the latest version from git tree. However, you may not want to mess with the packaging, so installing systemtap in your home directory is the fastest way to start using it. Assuming you have already installed make, gcc, g++, git-core, libebl-dev and sudo (just for convenience), type following commands:

git clone git://sources.redhat.com/git/systemtap.git
cd systemtap
STAP=$HOME/systemtap-git
./configure --disable-docs --prefix=$STAP && make && make install && \
            ( cd $STAP/bin && sudo chown root staprun && sudo chmod u+s staprun )
export PATH="$STAP/bin:$PATH"

To use stap as a normal user you must be in the stapdev group. So:

sudo groupadd -r stapdev
sudo usermod -aG stapdev $USER

Building a custom kernel suitable for SystemTap

Debian kernels do not contain debug information and there is no automatic way of installing or producing separate -dbg packages as RedHat has. Thus, there is no escape: you will have to manually configure and build a new kernel image package. It may help to lobby the Debian kernel-package maintainers, for example via debian bug #365349.

Now, get a kernel source package and kernel-package. I grabbed sid's 2.6.15 package...

apt-get install linux-source-2.6.15 kernel-package fakeroot

Configure the kernel source

Decompress your kernel source.

Just to avoid the hassle of configuring a clean kernel from scratch, copy a sane .config into your kernel source directory. A /boot/config-2.6.xxxxxxxx from a debian package will do just fine.

cd /usr/src
tar jxvf linux-source-2.6.15.tar.bz2
cd linux-source-2.6.15
cp /boot/config-2.6-xxxxxxxx .config

Configure your kernel

make menuconfig

Inside make configure, select/enable, in order, the following options:

Kernel hacking  --->
    [*] Debug Filesystem
    [*] Kernel debugging
        [*]   Compile the kernel with debug info
Instrumentation Support  --->
    [*] Kprobes (EXPERIMENTAL)

Optionally, for example if you need to use process.* probes, compile your kernel with CONFIG_UTRACE support:

[*] Infrastructure for tracing and debugging user processes  --->

• Notice: this option is available if you applied utrace patch.

Optionally, compile your kernel with CONFIG_RELAY support:

General setup  --->
    [*] Kernel->user space relay support (formerly relayfs)

• Notice: there is no "CONFIG_RELAY" in kernels older then 2.6.17. It seems that CONFIG_RELAY was added latter...

Configuring kernel-package

Add the following to to /etc/kernel-pkg.conf:

install_vmlinux = YES

Building your kernel

fakeroot make-kpkg --initrd --append-to-version=-systemtap-1.0 kernel_image kernel_headers

Notice the following things about the above command line:

--initrd

instruct kernel-package to build an initrd image

--append-to-version

will modify both the kernel name presented on uname -a and the Debian package name. I'm using -systemtap-1.0 here but feel free to change that. There are some restrictions on what this name can be, though. Read kernel-package documentation (README.gz) for more information on this.

kernel_image

well, we are building a kernel image package, aren't we?

kernel_headers

also build a kernel headers package to save time.

You can pass more options to make-kpkg. It should not be a problem here.

Starting with version 12.002, kernel-package has a debug target named kernel_debug. Using this target will generate a new -dbg package which will store the debug symbols. The new revised command now is:

fakeroot make-kpkg --initrd --append-to-version=-systemtap-1.0 kernel_image kernel_headers kernel_debug

Go get some coffee, read Slashdot, search for your advisor or what else because this will take some time...

Done? Now is the moment of truth, let's...

Install your custom kernel

New packages for your kernel (image and headers) were created. Install them:

dpkg -i ../kernel-image-2.6.15-systemtap-1.0_10.00.Custom_i386.deb
dpkg -i ../kernel-headers-2.6.15-systemtap-1.0_10.00.Custom_i386.deb

Note that since linux kernel 2.6.8, kernel-<type>-*.*.* packages have been renamed to linux-<type>-*.*.*

Add this new image to your bootloader list. Read lilo or grub documentation for more information on how to accomplish this.

Finishing up

• I didn't use to do the next steps but it seems that the combination "New SystemTap" + "Kernel Compilation in non-standard places" requires them.

We are almost there.

Make a copy of you build directory² to /lib/modules/<your new kernel version>/build. Yes, it's huge but yes, it has to be done.

Finally, check if there is a corresponding vmlinux image and a corresponding config file for your new kernel in /boot. If they don't exist, make sure that they do exist in your copy of the build directory.

Pause, breathe and reboot your box.

Enjoy

That's all. You can start stap'ing now

Footnotes