DTrace

DTrace is illumos’s dynamic tracing framework: a single tool that instruments the entire system — kernel and userland — safely enough to run on production. It was invented at Sun and is one of the reasons people run illumos at all. For a solnix user, DTrace is the observability answer to “what is this Nix-built binary actually doing on this machine, right now, without a rebuild.”

On illumos generally DTrace is mature, safe, and load-bearing. On solnix DTrace is inherited from the illumos gate; the one-liners below are real DTrace. Getting a full solnix system to the point where you are tracing your own services is gated on the bootstrap (see Introduction) — but the framework itself is not solnix’s to build.

What it is

DTrace exposes tens of thousands of instrumentation points (probes) across the kernel and userland. Every probe is dormant and costs nothing until you enable it. You write small scripts in the D language — a C-like language with awk-style pattern/action structure — that say “when this probe fires, in this context, do this.”

A probe is named by four parts:

provider:module:function:name

For example syscall::open:entry (the entry to the open syscall) or proc:::exec-success (a process successfully exec’d). Leaving a field blank matches all values, so syscall:::entry matches the entry of every syscall.

Providers

Providers are the sources of probes. The ones you reach for first:

The pid provider is how you trace your program’s functions with zero instrumentation compiled in — DTrace attaches to the live process.

A few one-liners

Which programs are being executed, system-wide:

pfexec dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

Count system calls by process name for a few seconds, then print a table:

pfexec dtrace -n 'syscall:::entry { @[execname] = count(); }'

Files being opened, and by whom:

pfexec dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'

A latency distribution (a quantize power-of-two histogram) of read syscalls:

pfexec dtrace -n 'syscall::read:entry { self->t = timestamp; }
                  syscall::read:return /self->t/ { @ = quantize(timestamp - self->t); self->t = 0; }'

Sample kernel stacks at 997 Hz to see where CPU time goes:

pfexec dtrace -n 'profile-997 { @[stack()] = count(); }'

Aggregations (@) are DTrace’s headline feature: they sum/count/quantize in the kernel and only ship the summary to userland, so tracing a busy system stays cheap.

Safety — why you can run this in production

DTrace was designed from the start to be safe on live systems, and this is not a footnote — it is the reason it exists:

• Probes are zero-cost when disabled and cheap when enabled.
• The D language has no loops and no arbitrary memory writes — a DTrace script cannot hang or corrupt the kernel.
• Accessing userland memory goes through safe copy-in (copyinstr, etc.) that faults gracefully instead of panicking.
• There are tunable limits on buffers and drops so a misbehaving script degrades (drops records) rather than harming the system.

This is what lets you point DTrace at a production process and ask hard questions without a maintenance window.

Why it matters for solnix

On a Nix system the software layer is immutable, reproducible store paths — great for building correctly, silent about runtime behavior on real hardware and real load. DTrace is the missing half: it tells you what a store-path binary does when it runs — which syscalls, which files, where the latency and CPU actually go — with no rebuild and no restart. It pairs naturally with MDB and core-file analysis for the post-mortem case.

Further reading

• dtrace(8) — command reference.
• Debugging — DTrace alongside MDB, core files, and crash dumps.
• Further reading & resources — the DTrace book and the one-liner collections.