Guardrail Tester

Guardrail tester

Permission rules match the text of a command, so git push origin +main, rm -r -f build, and bash -c '...' can walk past a deny rule written for the plain form. This skill checks the rules and hooks the user already has against about 90 dangerous commands in those forms, then replays their recent tool calls to count how often the rules would ask or block. The user gets a headline ("Your Claude Code guardrails block 58 of 90 dangerous commands outright. 24 more stop at a prompt..."), every miss with a tested fix, and their friction numbers. Everything is read and simulated locally; nothing is sent anywhere, and it runs the user's hook commands only after they say yes.

When to use

• The user asks whether their deny rules, ask rules, or hooks really stop a dangerous command.
• The user wants to test, audit, or find bypasses in their guardrails or permission settings.
• The user asks how many prompts their rules cause, or wants fewer without losing safety.
• The user just installed a guard (a hook, dcg, the repo's safe-settings template) and wants proof.

When not to use

• What the agent can reach on this machine (secret files, Docker, sudo): use sandbox-check.
• Turning a rule from AGENTS.md or CLAUDE.md into a hook: use rules-to-guards.
• Stopping a session that loops or overspends: use runaway-guard.
• Building a guard from scratch: recommend dcg or templates/claude-code-safe-settings/ in this repository, then test it here.

What the tester runs

Say this to the user in short before the first run with hooks. It is the whole contract.

• The battery commands never run. Each is only text inside the JSON a hook reads on stdin. The tester never runs them; a hook that runs or forwards its input would. Each battery command is also written to do nothing if run: its targets sit under a missing ./guardrail-tester-probe/ folder, its remotes and branches (probe-remote, probe-branch) are made up, and its hosts end in .invalid, a name reserved for addresses that resolve nowhere.
• Rules are simulated from each harness's documented matching rules, so results are labeled "simulated". references/harness-rules.md says what each harness simulation covers.
• It runs your hook commands only after you say yes (the --run-hooks flag). Each matching PreToolUse hook command then runs once per battery case, one run at a time, with a 10-second limit, from the folder its harness uses (usually the project), with the session's environment variables plus CLAUDE_PROJECT_DIR. A hook that writes a log or keeps state records those test calls. A hook that times out twice is not run again.
• Replay reads session transcripts on this machine, read-only, masks secrets in its output, and checks the rules only. With --replay-hooks (which needs --run-hooks) it also runs this project's hooks on replayed calls from this project. Calls from other project folders are checked against their own rules; their hooks never run.

scripts/battery.json holds the dangerous commands on purpose: they are the test. Each line carries the marker skillscan:allow, which tells this repository's security scanner that the line is test data, not a command the skill runs.

Steps

<skill-dir> means the folder that holds this SKILL.md (Claude Code shows it as the skill's base directory). Run every command from the user's project folder, and give each Bash call a 10-minute timeout (600000 ms): hooks run one at a time, so a slow hook makes the run long.

1. Run the rules-only test, which reads settings and runs no hook:

   bashCopy code
   python3 "<skill-dir>/scripts/test_guards.py" --project . --harness claude-code
   

   Add --harness for the harness you are running in (claude-code, codex, gemini-cli, opencode, or cursor) unless the user asks about all; without it the tester covers every harness with settings on this machine. Done when the output starts with a bold headline, or you have told the user the error (exit code 2 means a bad argument or an unreadable battery file).

2. Tell the user what was found: the settings files and every hook command from the report's "What was found" section, quoted exactly. Done when the user has seen the list.

3. Read each hook script, then ask before running hooks. Open the script each hook command runs. If a script runs, evaluates, or sends its input anywhere (eval, bash -c "$cmd", a curl with the input, a queue), or has other side effects such as writing a log, tell the user exactly that and recommend testing without hooks. Otherwise name the hook commands and say each will get test JSON for about 90 battery calls. On a clear yes, run this with a 10-minute Bash timeout:

   bashCopy code
   python3 "<skill-dir>/scripts/test_guards.py" --project . --harness claude-code --run-hooks --replay 500
   

   Add --replay-hooks only when the user also agrees that this project's hooks see up to 500 of their recent real calls. When the user declines, or a hook has side effects, run the same command without --run-hooks. Done when the headline matches the choice: it starts "Your ... guardrails block" after hooks ran, and "Without running your N hooks" when hooks exist and were skipped.

4. Pick the fixes. For each row in "Misses, worst first", take the Fix column: a deny rule the simulation proved catches that case and leaves everyday commands and files such as .env.example alone, or a named hook check (an extended regular expression printed below the tables). Use references/bypass-forms.md to explain why a form slips. Done when every miss you report has its fix or the note "use the sandbox".

5. Offer the change. Draft the settings edit or hook lines as a diff against the user's file, show it, and apply it only after a clear yes. Then rerun step 3 to show the new count.

Read the results

• Headline: how many battery commands are blocked outright; how many more stop at a prompt, with the number that ask only because of the permission mode in parentheses; how many run without asking, with the worst example. For Codex: how many its rules block and how many its sandbox stops or asks about. The replay friction comes last when replay ran.
• Table columns: Blocked; Not blocked; Asks (rule, hook, or check), prompts that still appear in auto mode; Asks (mode only), prompts that auto and bypassPermissions modes drop; Runs without asking, which counts allow rules, the read-only set, sandbox auto-allow, and auto mode's classifier.
• Mode: with no defaultMode in the settings, the tester simulates auto mode, the built-in default on Claude Code 2.1.283 and later, and adds a "Claude Code in Manual mode" row. --mode simulates another mode. Plan mode lasts only until a plan is approved.
• Should: block cases count as stopped only when blocked outright, because people approve prompts by reflex. ask cases count when they prompt or are blocked. --fail-on-miss uses this count.
• What happens: blocked, asks first, runs without asking, left to the auto-mode classifier, or unknown. The part in parentheses names the layer that decided: a hook, a rule, a built-in check (protected paths, critical-path removals, the read-only command set), the mode, or the sandbox. With a sandbox on, a command that needs the network or writes outside the project shows asks first (needs network) or stopped by the sandbox.
• Unknown: a hook did not answer within the tester's limit while the harness would wait longer. Unknown cases are not counted as misses; raise --hook-timeout to settle them.
• Misses are listed worst first: runs without asking, then asks only because of the mode, then asks because of a rule, hook, or check.
• Friction: replayed calls that would ask or be blocked, each simulated in the permission mode its session recorded, and real dangerous calls (matched by the hook checks) that ran and would still not be blocked as expected. A long command shows as the hook checks it matches.
• Configuration problems: for example a hook that exits 1 (the call goes ahead), a hook with no timeout, a Bash matcher that Cursor never fires, a Codex hook nobody trusted in /hooks, a shadowed OpenCode deny, or bypass mode left available.
• Notes on single cases: where Claude Code might apply a Read rule the docs do not extend to a command. --json has every case with verdict, bucket, decided_by, rules_verdict, and fix, plus hook_checks, every hook check by name.

Report to the user

1. The headline, verbatim, in bold.
2. The misses table, cut to its first eight rows (the report lists the worst first), each with its fix.
3. The friction line and up to three dangerous replayed calls, quoted as the report prints them.
4. Configuration problems, high first, one line each.
5. The two or three fixes that close the most misses, then the offer from step 5. Point to dcg and templates/claude-code-safe-settings/ rather than writing a new guard.

Files

• scripts/test_guards.py: the tester. Flags: --harness, --project, --run-hooks, --no-hooks (the default), --hook-workers N (default 1), --replay N, --replay-hooks, --since DAYS, --mode, --battery, --hook-timeout, --home, --json, --out, --fail-on-miss (exit 1 when a case is not blocked as expected).
• scripts/battery.json: the dangerous commands, one case per line, each with what it needs to do its harm (network, write-outside, write-git, write-project).
• scripts/claude_rules.py, scripts/harness_rules.py: rule simulation per harness.
• scripts/hook_runner.py: runs one hook command with test JSON and reads its decision.
• scripts/shell_split.py: splits a command line into the commands it runs.
• scripts/transcripts.py: reads session transcripts for replay (shared copy).
• scripts/safe.py: masks secrets and puts text from settings, hooks, and transcripts in inline code for the report (shared copy).
• references/bypass-forms.md: each form in the battery and why rules miss it.
• references/harness-rules.md: matching rules per harness, with sources and the date checked.