Debugging

illumos has a deep, unusually coherent debugging toolkit — crash dumps, the modular debugger mdb, DTrace, core files with CTF type data, and the fault management architecture — and solnix inherits all of it from the gate. This page walks the common cases: a service that will not start, a program that crashed, and a system that panicked or hung. It is adapted from the illumos user guide and framed for a solnix user.

Honest status. solnix is early — only a preview image exists so far (see Introduction), so end-to-end “debug your live solnix box” is not yet an everyday thing. The commands below (svcs, mdb, savecore, pstack, coreadm, dumpadm, fmadm) are the real illumos commands and work on any illumos host. Where a step is solnix-specific and not built yet, it says so.

Note: commands that need privilege are shown with pfexec (illumos’s RBAC-backed sudo-equivalent). On some systems sudo is used instead.

A service failed to start

This is the most common everyday case and the fastest to diagnose. If something that should be running is not, ask SMF why:

svcs -x

svcs -x walks the service dependency graph and prints, for each broken service, the state, the likely cause, and the path to its log. Then read that log:

svcs -l svc:/network/ssh:default          # find the logfile: line
tail -50 /var/svc/log/network-ssh:default.log

If the service is in maintenance because it kept failing, fix the cause and retry:

pfexec svcadm clear svc:/network/ssh:default

See SMF services for the full state model and command set.

A program crashed (core files)

When a userland process crashes, illumos can save a core file — a snapshot of the process’s memory — that you inspect after the fact.

Check the core configuration first:

coreadm

     global core file pattern:
     global core file content: default
       init core file pattern: core
            global core dumps: disabled
       per-process core dumps: enabled

If per-process core dumps are enabled and the global pattern is empty, a crash writes a file named core in the process’s working directory. To set a predictable global location and pattern:

pfexec coreadm -g /var/cores/core.%f.%p -e global -e per-process
# %f = executable name, %p = pid; see coreadm(8) for all specifiers

Read the core file

The quick answer — the stack at the point of the crash:

pstack core

The deep answer — open it in the modular debugger:

mdb core
> ::status          # summary: signal, faulting instruction
> $C                # C stack backtrace
> ::stack           # same, another form
> ::regs            # registers at the fault
> $q                # quit

Because illumos binaries carry CTF type data, mdb can print structures with their real field names and types — you are not staring at raw hex. On solnix the binary is a /nix/store path; mdb resolves its symbols and CTF the same way it does for any illumos binary, and the store path pins the exact build that produced the core.

The system panicked or hung (crash dumps)

When the kernel panics, illumos saves a crash dump of kernel memory to the dump device, then reboots. On the next boot you extract and analyze it.

Check the dump configuration:

pfexec dumpadm

      Dump content: kernel pages
       Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash/<hostname>
  Savecore enabled: yes
   Save compressed: on

Extract the dump after the reboot:

pfexec mkdir -p /var/crash/`hostname`
pfexec savecore
cd /var/crash/`hostname`

savecore writes unix.<n> (the kernel) and vmcore.<n> (memory). Then pull the essential post-mortem information into one file with mdb:

echo '::panicinfo
::cpuinfo -v
::threadlist -v 10
::msgbuf
*panic_thread::findstack -v
::stacks' | mdb <n> > ~/crash.<n>

Keep crash.<n> (and, if you can, the dump itself) for a bug report. Note that a crash dump can contain confidential in-memory data — use judgment before sharing it publicly.

Forcing a dump from a hung system

If the system is wedged but responsive enough:

pfexec savecore -L      # live dump without rebooting
# or
pfexec reboot -d        # reboot and force a dump

For a truly frozen box, illumos supports dropping into the kernel debugger (kmdb) via the boot loader or an NMI; that is advanced and hardware-specific — see the illumos user guide linked below.

Hardware and diagnosed faults (FMA)

The fault management architecture turns error telemetry into diagnosed faults:

fmadm faulty           # currently diagnosed faults and affected components
fmdump                 # the fault log
fmdump -v              # verbose, with the fault event detail
fmstat                 # fault-manager module statistics

Each fault carries a stable message ID (like ZFS-8000-8A) you can look up. FMA covers hardware (CPU, memory, disks) and some software faults, and it hands service faults to SMF — so fmadm faulty and svcs -x together cover “what is wrong with this machine.”

Live tracing (DTrace)

For behavior on a running system — not a crash — reach for DTrace: which syscalls a process makes, where latency comes from, what is burning CPU, without a rebuild or restart. See DTrace.

Reporting a bug to solnix

When you hit something you believe is a solnix (not upstream illumos) problem, include:

what you did and what you expected versus what happened;
the solnix generation / boot environment (beadm list) so the exact build is identifiable;
the relevant artifact: svcs -x output and the service log for a service failure; pstack/mdb output for a crash; the crash.<n> file for a panic; fmadm faulty for a hardware fault;
the store path(s) involved — a /nix/store path pins the exact build, which is the whole point of a reproducible system.

Send it to the development list — see Contributing for the current channels (hackers@solnix.io, planned). If the bug reproduces on stock illumos with no Nix involved, it likely belongs upstream at illumos.org.

The solnix handbook