Detection engineering
A detection rule is a claim: this pattern means an attack. The claim is worth nothing until it has been tested both ways.
Every rule must pass two tests, and neither substitutes for the other:
- Fires on known-bad — it actually triggered on a true-positive sample.
- Silent on known-good — it did not trigger on a benign sample.
A rule tested only against the attack is unproven against false positives. A rule never tested against the attack is unproven, period. If you cannot show both for a rule in production, you have a rule collection, not a detection programme.
When to Use
- Writing a Sigma/Suricata/YARA rule, reviewing a detection before deployment, triaging alert fatigue, or deciding whether a rule has earned production.
- Use only within explicit authorization and the stated rules of engagement.
When NOT to Use
- Do not treat this skill as permission to access, test, or change a target.
- Do not convert missing evidence into a clean result or a stronger claim.
Prove it before it ships
shuv run {baseDir}/scripts/check_detection.py record.json
Takes a test record per rule and fails any rule missing either proof, firing on
known-good, over the false-positive ceiling, or unmapped to an ATT&CK
technique. Exit 0 only when every rule is proven both ways; 1 on any
unproven rule; 2 on empty input. See references/test-record.md.
Production decision
| Known-bad result | Known-good result | Disposition |
|---|---|---|
| Fires | Silent | Eligible for controlled rollout |
| Fires | Fires | Rewrite or narrowly tune; false-positive proof failed |
| Silent | Silent | Do not ship; the rule did not detect the behavior |
| Missing | Any result | Unproven; collect the missing test evidence |
| Fires | Silent, but provenance or mapping missing | Hold until the rule can support a defensible coverage claim |
Record the decision per rule. A passing rule does not prove that the production
pipeline will deliver or alert on it; live validation belongs to
detection-validate.
False positives are the silent killer
Analysts go alert-blind within weeks when most of what fires is garbage — then the real alert gets the same shrug. Manage it in this order:
- Baseline first. Run a new rule in detection-only mode for ~2 weeks. Count the fires, investigate a sample, learn what normal looks like.
- Exception lists, not rule disablement. If a rule is noisy on one service account, exclude that account — do not switch the rule off for everyone.
- Track the rate. Above ~90% false positives the logic is wrong, not the exceptions. Rewrite it; do not tune it.
Every rule carries its provenance
For each rule you should be able to answer: where did the idea come from, what threat it addresses, what log source powers it, when it was last tested, and its false-positive rate. A rule whose last test date is unknown is an unproven rule regardless of how long it has been deployed — time in production is not evidence.
Keep detections in version control
Rules in Git, tests beside them, CI that validates syntax and runs each rule against sample known-bad and known-good log data before it compiles to your SIEM format. A detection pipeline without a test step ships unproven rules by construction.
Rationalizations to reject
| Rationalization | Why it fails |
|---|---|
| “The attack sample fired, so the rule can ship.” | It remains unproven against false positives; the SOC goes blind |
| “It has been in production for a year, so it works.” | Time deployed is not a test. If the last-tested date is unknown, the rule is unproven |
| “The rule is noisy, so disable it.” | Disabling it loses the true positives with the false ones; use a bounded exception instead |
| “We can tune around a 95% false-positive rate.” | The logic is wrong, not the thresholds — rewrite it |
| “ATT&CK mapping is optional documentation.” | Without the mapping, the rule cannot support a defensible coverage claim |
References
- Test record format — the JSON the checker consumes, and what each field proves
