Detection validation
Purple team answers one question, per technique: we executed it — did a detection fire? Everything else is commentary.
"We think we're covered" is not coverage. Coverage is a specific claim about specific techniques, backed by a recorded outcome for each. A map with blanks in the detection column is the same lie as a port scan that came back clean with zero coverage — a confident summary over an unanswered question.
When to Use
- Running a purple-team exercise, measuring detection coverage, validating that a SOC can see a technique, or turning 'we think we're covered' into evidence per technique.
- Use only within explicit authorization and the stated rules of engagement.
When NOT to Use
- Do not treat this skill as permission to access, test, or change a target.
- Do not convert missing evidence into a clean result or a stronger claim.
Every executed technique gets exactly one honest outcome
| Outcome | Meaning |
|---|---|
covered | executed, a detection fired |
partial | fired weakly, or on only some variants |
not-detected | executed, nothing fired — a real gap |
not-detectable | passive technique with no observable (OSINT, reading public records) — an accepted gap, not a failure |
unknown | executed but the outcome was never recorded — the one that poisons the report |
The distinction that matters: not-detectable is honest (you looked, there is
nothing to see, you accept it) while unknown is a blank wearing a confident
coverage number. They must never be conflated.
Score the map
shuv run {baseDir}/scripts/coverage_map.py map.json
Computes real coverage as covered+partial over everything executed that could
be detected — passive not-detectable techniques are removed from the
denominator. Exit 0 only when every executed technique has a known, non-gap
outcome; 1 when any not-detected or unknown remains; 2 on bad or empty
input. See references/coverage-map.md.
Validation record
For each technique, return the technique and variant, execution receipt, expected observable, telemetry-health evidence, alert evidence, one outcome from the table above, limitations, and an owner for any gap. Keep the execution receipt and alert receipt separate: the same timestamp or operator note cannot prove both sides of the join.
End with counts for covered, partial, not-detected, not-detectable, and
unknown, plus the denominator used. Never publish only a percentage.
Pair with the attacker and the detector
This skill is the join between offense and defense:
- The offensive technique comes from the operator (or an attack simulation). If
you cannot execute it, you cannot validate its detection — an untested
technique is
unknown, never assumecovered. - The detection that should fire is engineered and proven separately. A rule that passed its own true-positive test in isolation can still fail to fire in a live exercise against the real technique variant — that is exactly what this measures, and why the exercise is not redundant with the rule's unit test.
Rationalizations to reject
| Thought | Reality |
|---|---|
| "We ran 40 techniques, we're covered." | Not until all 40 have a recorded outcome. Blanks are unknown, and unknown is not coverage. |
| "No alert fired, so the technique must be benign." | No alert fired means you did not detect it. That is a gap, not an all-clear. |
| "It's probably detected, the rule exists." | A rule existing is not a rule firing. The exercise exists because rules fail live that passed in isolation. |
| "Mark the passive stuff as covered so the number looks complete." | not-detectable is removed from the denominator, not counted as covered. Honest beats complete. |
| "We'll fill in the blanks later." | A blank is unknown now. Ship the map honest with the gaps visible, or re-run the technique. |
| "100% coverage." | Almost always means the un-run techniques were left off the map. Coverage is over what you executed, and the map must show what you did not. |
References
- Coverage map format — the JSON the scorer consumes, the five outcomes, and how the denominator is computed
