Homelab triage
Purpose: quickly localize a problem to a layer, then hand off to the specific
skill. Do not fix blindly from here. Localize first. The platform catalogs under
capabilities/ are not loaded up front; the skill you hand off to reads the one
it needs.
Step 1: get the wide view
Run both sweeps. They are read-only and safe.
hh overview # vitals: uptime, memory, root disk, pool verdict hh inventory # what is RUNNING: VMs, LXCs, containers, apps
Run hh inventory too, not just hh overview. Overview reports vitals only and
lists no guests on any platform, so a VM or LXC that is stopped, crashed, or
failed to start after an upgrade is invisible to it. A "state of the homelab"
answer is not complete until guest state is included for BOTH hypervisor-ish
hosts: Proxmox (qm list VMs, pct list LXCs) and TrueNAS (VMs and LXCs via
virt.instance.query on 25.04/25.10, vm.query on 24.10 and 26). hh inventory
already does both automatically.
Call out anything not running by name. A stopped guest is a finding, not background noise, especially on a TrueNAS host recently upgraded to 26, where the Incus-to-libvirt migration is known to leave instances behind (see the gotchas in infra/truenas.md).
If you need the raw host list first, hh list shows every registered host. To
spot-check one, hh run <alias> "uptime; df -h /".
Health check: never green without the kernel log
hh overview and the pool/SMART/alert views miss a class of hardware and kernel
faults - PCIe AER, NIC link flaps, ATA/disk resets, MCEs, and OOM kills. Before
you report the homelab (or any TrueNAS host) healthy, scan the kernel ring
buffer:
hh run truenas "journalctl -k -b -p warning --no-pager | grep -viE 'veth|br-[0-9a-f]{12}' | tail -40"
Empty output is the pass. Any AER / MCE / ATA-reset / link-down / OOM line is a real finding - localize it (Step 3) before declaring green. Same principle for other hosts: a clean overview is not a clean dmesg. See "Health check must-dos" in infra/truenas.md for the full rationale.
Step 2: check reachability and DNS early
A surprising share of "everything is broken" reports are network, not the apps.
ping -c2 <each host mesh ip> hh run <alias> "echo ok" netbird status
If a host answers on LAN but not on its mesh IP, treat it as a mesh problem and go to network-diag, not the host skill.
Step 3: localize by layer
Walk the escalation ladder from CLAUDE.md and pick the matching skill:
- A single app or download is broken, hosts fine -> docker-stack-ops
- A whole VM or container host is off or thrashing -> proxmox-ops
- Storage errors, pool degraded, dataset missing, disk failing -> truenas-ops
- Broad unreachability, DNS, mesh, tunnels, switch -> network-diag
- "What is running / what changed" across the estate -> inventory skill; run
hh diffto see exactly what changed since the last saved snapshot (a recent change is often the cause) - Need to restore, roll back, or verify a backup exists -> backup-restore
Not a fault but a "is this exposed / are we patched" question -> security-audit.
Step 4: report before acting
Summarize what you found in plain terms: which layer, which host, the evidence, and the single most likely cause. Propose the next diagnostic or the fix, and wait for a go-ahead before anything state-changing.
Step 5: after resolution
If this turned into a real incident, append a runbook entry (see
runbooks/README.md) so the next occurrence is faster.
