systemtap

DTrace

LTTng

project

license

GPL

CDDL

GPLv2

operating system support

Linux

Solaris, Mac OS X, BSD, QNX

Linux

processor support

as per kprobes: x86, x86_64, ppc64, ia64, s390, arm, sparc?

x86, x86_64, SPARC, ppc, ppc64

x86, x86_64, SPARC, SPARC64, ppc, ppc64, sh, sh64, ia64, s390, MIPS 32/64, ARM, (arch-agnostic core)

kernel coupling - interdependent development/schedule

little

lot

lot

core developers

open community

open community

open community

development began

January 2005

October 2001

January 2005

development status

ongoing

stable with continuing development

ongoing

target audience

developers, users, sysadmins

developers, users, sysadmins

userspace and kernel developers, users, sysadmins

target usage

symbolic debugging, tracing, profiling

debugging, tracing, profiling

debugging, tracing, profiling, monitoring

language

style

scripting

scripting

hard-coded kernel C code

full control structures (conditionals, loops, functions)

yes

no

yes

variable typing

implicit, inferred

implicit

explicit

complex reporting (join/projection/select)

yes (first principles, iteration, conditionals)

limited (with printa())

n/a

scalable aggregates

yes

yes

n/a

aggregate value readable by script

yes

no

n/a

thread-local variables

yes (from first principles via tid-indexed auxiliary arrays)

yes (implemented efficiently)

n/a

speculative tracing

yes (from first principles via auxiliary data and control structures)

yes

n/a

binary tracing record

yes

?

yes

early boot tracing

not yet [[http://sourceware.org/PR2035]

yes

not "as early as it could" yet

probe execution

optimized native code

interpreted bytecodes

optimized native code

probing capability

number of available symbolic probe points in the kernel

millions (statements, tracepoints, markers)

thousands (functions, markers)

thousands (functions, tracepoints, markers)

number of available symbolic probe points in user-space

zillions (statements, functions)

millions (functions, markers)

in progress http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html

high-speed breakpoint-less userspace data extraction

no

no

in progress http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html

probe arbitrary statements in code symbolically (function entry, exit, interior, source code co-ordinates)

yes (using debugging information)

limited to ABI boundaries (function entry/exit, markers)

no

symbolically extract arbitrary data at probe point

yes (any context-visible variable as preserved by compiler)

limited (all data visible; local variables may be accessed by using manual register offsets)

no

extraction of data by static instrumentation

yes (with help of markers)

yes (with markers)

yes (with tracepoints and markers)

non-symbolic (manually addressed) probe points / data

yes

yes

no

probe dynamically loaded kernel objects

yes

yes

yes

concurrent probes on multiprocessors

yes

yes

yes

context pointer type punning/casting

yes

yes

yes

statically inserted probe points, kernel side

yes (markers)

yes (SDT)

yes (tracepoints, markers)

end-user extendable probe library

yes (script based tapsets)

no

yes

trace user-space stack backtraces

yes

yes (ABI helps)

had this feature previously, not currently

statically inserted probe points, user side

yes <sdt.h>

yes (USDT)

in progress http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html

trace Java programs

yes

yes

had previously http://ltt.polymtl.ca/svn/trunk/obsolete/ltt-usertrace/java/, not currently

trace Java stack backtraces

no

yes

no

statically inserted probe points, Java

from JVM

yes

no yet

trace script language programs

yes, distributions to activate

yes: Ruby, JavaScript, Perl, Python, PHP, APL, Bourne shell, ksh, zsh, Tcl

yes, through debugfs file

timer-based probing

yes (profiling interrupt, software timers)

yes

no

hardware performance counter based probing

soon http://sourceware.org/PR909

yes

soon

safety

protected probe execution environment

yes

yes

no (hand-written)

time-limited probe handler execution

yes (statement counting)

yes (built into interpreter)

no

non-blocking, atomic probe handlers

yes

yes

yes

space-limited execution

yes (static allocation of all data)

yes

yes (bounded to buffer size + probe size + memory allocated by probe)

division-by-zero protection

yes

yes

no

null pointer dereferencing protection

yes

yes

no

means available to bypass protection for advanced users

yes (guru mode, embedded-C)

limited (predefined destructive actions for well-defined modifications)

Optimized probes shipped with mainline kernel, variable argument list (printf-like) type safe data serializer available for ad-hoc use

safe use on production systems

soon

yes

yes

translate-time error checks

many

many

n/a

run-time error checks performed by

automatically generated C code

bytecode interpreter

community review process

use by unprivileged users

developer vs. user distinction

graduated access by privilege level

not currently

misc.

analysis performed

online

online

offline (post-mortem)

built-into the system kernel

not necessary

yes

yes (mainlining in progress)

lockless tracing

no

yes

yes

zero-copy trace data extraction

?

no (designed to allow for concise traces)

yes

provides analysis tools to navigate in large multi-GB traces

no

no (designed to allow for concise traces)

yes (lttv)