• DTrace
• One-Liners

The following library of DTrace one-liners were last tested on FreeBSD 10.0. In FreeBSD 12.0, dwatch was added to the base.

Version

# Show dtrace version
dtrace -V

# Show dwatch version
dwatch -V

Listing Probes

# List probes and search for string "foo":
dtrace -l | grep foo

# Summarize probes by providers:
dtrace -l | awk '{ print $2 }' | sort | uniq -c | sort -n

# List probes for a particular provider:
dtrace -l -P syscall

Using dwatch:

# List probes and search for string "foo":
dwatch -lr foo

# List unique providers:
dwatch -lP

# List probes for a particular provider:
dwatch -lP syscall

# List unique functions:
dwatch -lf

# List probes for functions ending in read:
dwatch -lf '*read'

# List probe-tuples ending in read using regex:
dwatch -lr 'read$'

# Watch probe traversal for a given command:
dwatch -F 'pid$target:::' -c true

Syscalls

# Trace file opens with process and filename:
dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'

# Count system calls by program name:
dtrace -n 'syscall:::entry { @[execname] = count(); }'

# Count system calls by syscall:
dtrace -n 'syscall:::entry { @[probefunc] = count(); }'

# Count system calls by syscall, for PID 123 only:
dtrace -n 'syscall:::entry /pid == 123/ { @[probefunc] = count(); }'

# Count system calls by syscall, for all processes with a specific program name ("nginx"):
dtrace -n 'syscall:::entry /execname == "nginx"/ { @[probefunc] = count(); }'

# Count system calls by PID and program name:
dtrace -n 'syscall:::entry { @[pid, execname] = count(); }'

# Find the parent of a process calling a syscall
dtrace -n 'syscall::read:entry { printf("execname=%s parent_execname=%s", execname, curthread->td_proc->p_pptr->p_comm); }'

# Summarize requested read() sizes by program name, as power-of-2 distributions (bytes):
dtrace -n 'syscall::read:entry { @[execname] = quantize(arg2); }'

# Summarize returned read() sizes by program name, as power-of-2 distributions (bytes or error):
dtrace -n 'syscall::read:return { @[execname] = quantize(arg1); }'

# Summarize read() latency as a power-of-2 distribution by program name (ns):
dtrace -n 'syscall::read:entry { self->ts = timestamp; } syscall::read:return /self->ts/ {
    @[execname, "ns"] = quantize(timestamp - self->ts); self->ts = 0; }'

# Summarize read() latency as a linear distribution (0 to 1000, step 5) by program name (ms):
dtrace -n 'syscall::read:entry { self->ts = timestamp; } syscall::read:return /self->ts/ {
    @[execname, "ms"] = lquantize((timestamp - self->ts) / 1000000, 0, 1000, 5); self->ts = 0; }'

# Summarize read() on-CPU duration as a power-of-2 distribution by program name (ns):
dtrace -n 'syscall::read:entry { self->ts = vtimestamp; } syscall::read:return /self->ts/ {
    @[execname, "ns"] = quantize(vtimestamp - self->ts); self->ts = 0; }'

# Count read() variants that "nginx" is using (if previous one-liners didn't work):
dtrace -n 'syscall::*read*:entry /execname == "nginx"/ { @[probefunc] = count(); }'

# Summarize returned pread() sizes for "nginx" as distributions (bytes or error):
dtrace -n 'syscall::pread:return /execname == "nginx"/ { @ = quantize(arg1); }'

# Count socket accept() variants by process name:
dtrace -n 'syscall::*accept*:return { @[execname] = count(); }'

# Count socket connect() variants by process name:
dtrace -n 'syscall::*connect*:return { @[execname] = count(); }'

# Summarize returned pread() sizes for "nginx"... and label the output:
dtrace -n 'syscall::pread:return /execname == "nginx"/ { @["rval (bytes)"] = quantize(arg1); }'

Using dwatch:

# Trace file opens with process and filename:
dwatch -X open

# Find the parent of a process calling a syscall
dwatch -R syscall::read:entry

# Watch syscall probe traversal:
dwatch -F syscall

# Watch syscall traversal by ruby processes:
dwatch -k 'ruby*' -F syscall

# Watch syscall traversal by processes containing `daemon' in their name:
dwatch -k '*daemon*' -F syscall

# Watch open and openat syscalls until pid 1234 exits:
dwatch -X open -- -p 1234

Process Tracing

# Trace new processes showing program name (and args if available):
dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

# Count process-level events:
dtrace -n 'proc::: { @[probename] = count(); }'

# Find the kernel function returning EINVAL for a given process
dtrace -n 'fbt:::return /(int)arg1 == EINVAL && pid == $pid/ { stack(); }' -c ./a.out

Using dwatch:

# Trace new processes showing program name (and args if available):
dwatch exec-success

# Trace new processes showing parent program and command executed:
dwatch execve

# Trace new processes showing mini process-tree for command executed:
dwatch -R execve

# Trace new processes belonging to root super-user:
dwatch -u root execve

# Trace new processes belonging to wheel super-group:
dwatch -g wheel execve

# Trace new processes belonging either to user daemon or nobody:
dwatch -u '1|65534' execve

# Trace new processes belonging either to group daemon or nobody:
dwatch -g '1|65534' execve

# Trace new processes running inside jail named myjail:
dwatch -j myjail execve

# Trace new processes running unjailed (jailed processes ignored):
dwatch -j 0 execve

# Trace new processes forked by pid 1234:
dwatch -p 1234 execve

# Trace new processes forked by either pid 1234 or a child of pid 1234:
dwatch -R -p 1234 execve

# Trace new processes forked by either pid 1234 or 5678:
dwatch -p '1234|5678' execve

# Trace new processes matching either mkdir or rmdir using regex:
dwatch -z '(mk|rm)dir' execve

Signals

Using dwatch:

# Watch signals being passed to kill syscall:
dwatch -X kill

# Watch signals being passed between bash(1) and vi(1):
dwatch -k bash -k vi -X kill

# Display the first process to call kill syscall and then exit:
dwatch -N 1 kill

Profiling

# Count sampled thread names on-CPU at 997 Hertz:
dtrace -n 'profile-997 { @[stringof(curthread->td_name)] = count(); }'

# Count sampled non-idle thread names on-CPU at 997 Hertz:
dtrace -n 'profile-997 /!(curthread->td_flags & 0x20)/ { @[stringof(curthread->td_name)] = count(); }'

# Count sampled on-CPU kernel stacks at 99 Hertz:
dtrace -n 'profile-99 /arg0/ { @[stack()] = count(); }'

# Count sampled process names and on-CP user stacks at 99 Hertz:
dtrace -n 'profile-99 /arg1/ { @[execname, ustack()] = count(); }'

Using dwatch:

# Trace statfs for 5 minutes and then exit:
dwatch -T 5m statfs

Storage I/O

# Count kernel stacks leading to block device I/O:
dtrace -n 'io:::start { @[stack()] = count(); }'

Using dwatch:

# Trace kernel stacks leading to block device I/O:
dwatch -X io-start

Scheduler

# Count kernel stacks leading to a context-switch off-CPU:
dtrace -n 'sched:::off-cpu { @[stack()] = count(); }'

Using dwatch:

# Watch processes entering system CPU scheduler:
dwatch on-cpu

# View the first 100 scheduler preemptions with less:
dwatch -y -N 100 preempt | less -R

IP

# Count IP-level events:
dtrace -n 'ip::: { @[probename] = count(); }'

Using dwatch:

# Trace IP-level events:
dwatch -X ip

UDP

# Count UDP-level events:
dtrace -n 'udp::: { @[probename] = count(); }'

Using dwatch:

# Trace UDP sent messages by remote IP address:
dwatch -X udp-send

# Trace UDP received messages by remote IP address:
dwatch -X udp-receive

TCP

# Count TCP-level events:
dtrace -n 'tcp::: { @[probename] = count(); }'

# Trace TCP accepted connections by remote IP address:
dtrace -n 'tcp:::accept-established { trace(args[3]->tcps_raddr); }'

# Count TCP passive opens by remote IP address:
dtrace -n 'tcp:::accept-established { @[args[3]->tcps_raddr] = count(); }'

# Count TCP active opens by remote IP address:
dtrace -n 'tcp:::connect-established { @[args[3]->tcps_raddr] = count(); }'

# Count TCP sent messages by remote IP address:
dtrace -n 'tcp:::send { @[args[2]->ip_daddr] = count(); }'

# Count TCP received messages by remote IP address:
dtrace -n 'tcp:::receive { @[args[2]->ip_saddr] = count(); }'

# Summarize TCP sent messages by IP payload size, as a power-of-2 distribution:
dtrace -n 'tcp:::send { @[args[2]->ip_daddr] = quantize(args[2]->ip_plength); }'

Using dwatch:

# Trace TCP accepted connections by remote IP address:
dwatch -X tcp-accept-established

# Trace TCP active opens by remote IP address:
dwatch -X tcp-connect-established

# Trace TCP sent messages by remote IP address:
dwatch -X tcp-send

# Trace TCP received messages by remote IP address:
dwatch -X tcp-receive

# Trace TCP activity while given nc command runs:
dwatch -X tcp -- -c "nc -zvw10 google.com 22"

Kernel Locks

# Sum kernel adaptive lock block time by process name (ns):
dtrace -n 'lockstat:::adaptive-block { @[execname] = sum(arg1); }'

# Summarize adaptive lock block time distribution by process name (ns):
dtrace -n 'lockstat:::adaptive-block { @[execname] = quantize(arg1); }'

# Sum kernel adaptive lock block time by kernel stack trace (ns):
dtrace -n 'lockstat:::adaptive-block { @[stack()] = sum(arg1); }'

# Sum kernel adaptive lock block time by lock name (ns):
dtrace -n 'lockstat:::adaptive-block { @[arg0] = sum(arg1); } END { printa("%40a %@16d ns\n", @); }'

# Sum kernel adaptive lock block time by calling function (ns):
dtrace -n 'lockstat:::adaptive-block { @[caller] = sum(arg1); } END { printa("%40a %@16d ns\n", @); }'

Namecache

# Count namecache lookups by program
dtrace -n 'vfs:namecache:lookup: { @missing[execname] = count(); }'

# Count namecache misses by program
dtrace -n 'vfs:namecache:lookup:miss { @missing[execname] = count(); }'

VFS

Using dwatch:

# Trace VFS lookup events by path:
dwatch -X vop_lookup

# Trace VFS lookup (VOP_LOOKUP(9)) paths containing /lib/:
dwatch -X vop_lookup -r /lib/

# Trace VFS create events by path:
dwatch -X vop_create

# Trace VFS remove events by path:
dwatch -X vop_remove

# Trace VFS symlink events by link and path:
dwatch -X vop_symlink

# Trace VFS readdir events by path:
dwatch -X vop_readdir

# Trace VFS mkdir events by path:
dwatch -X vop_mkdir

# Trace VFS rmdir events by path:
dwatch -X vop_rmdir

# Trace VFS rename events by path and destination:
dwatch -X vop_rename

# Trace VFS mknod events by path:
dwatch -X vop_mknod

Raw Kernel Tracing

# Count kernel slab memory allocation by function:
dtrace -n 'fbt::kmem*:entry { @[probefunc] = count(); }'

# Count kernel slab memory allocation by calling function:
dtrace -n 'fbt::kmem*:entry { @[caller] = count(); } END { printa("%40a %@16d\n", @); }'

# Count kernel malloc() by calling function:
dtrace -n 'fbt::malloc:entry { @[caller] = count(); } END { printa("%40a %@16d\n", @); }'

# Count kernel malloc() by kernel stack trace:
dtrace -n 'fbt::malloc:entry { @[stack()] = count(); }'

# Summarize vmem_alloc()s by arena name and size distribution:
dtrace -n 'fbt::vmem_alloc:entry { @[args[0]->vm_name] = quantize(arg1); }'

# Summarize TCP life span in seconds:
dtrace -n 'fbt::tcp_close:entry { @["TCP life span (seconds):"] =
    quantize((uint32_t)(`ticks - args[0]->t_starttime) / `hz); }'

Using dwatch:

# Trace processes performing writes with small buffers:
dwatch -t 'arg2<10' -E 'printf("%d",arg2)' write

# Similar to above but ignore dtrace executable:
dwatch -X write -t 'execname != "dtrace" && this->nbytes < 10'

CategoryDtrace CategoryHowTo