Harness Feedback
Treat "the harness is too strict" as an engineering finding, not as permission to disable a safety check. Find the boundary that owns the mismatch and move the check to the narrowest profile that actually needs its evidence.
Profiles
Use these profiles unless the project has a more specific, documented contract:
| Profile | Purpose | Typical blocking checks |
|---|---|---|
staging-smoke | Fast proof that the changed build starts and the critical path works | build, focused regression, one stable smoke/contract check |
security-proof | Prove an adversarial or trust-boundary claim | hostile tests, source/collector proof, fresh-context evaluator |
release-attestation | Prove the exact releasable artifact and its identity | signing, Authenticode/tool identity, installer/package checks |
nightly-stress | Find intermittent and capacity failures | race, stress, AV/OS matrix, long-running evals |
staging-smoke must not require signing, production credentials, a release
certificate, or a long VM stress run. security-proof may run on an unsigned
staging build when its claim is source or runtime behavior. A release check may
remain blocking for release promotion without becoming a per-edit gate.
Feedback Loop
For every overload signal, record:
- requested profile and change boundary;
- gate that blocked or dominated the run;
- command, elapsed time, failure count, and evidence actually produced;
- whether the gate was relevant, duplicated, flaky, or misplaced;
- the smallest profile split or deletion of duplicate coverage;
- a before/after run of the affected profile and a fresh review of the rule.
Use the deterministic harness-load-advisor.py signal as an intake event. It
stores metadata outside the repository and forces the final report to name the
mismatch. Durable policy changes belong in Git; raw session traces do not.
Required Report
Do not write "overkill" and move on. Report:
textHarness feedback: OVERLOAD | CLEAR Requested profile: staging-smoke | security-proof | release-attestation | nightly-stress Mis-scoped gate: <name> Evidence: <command, result, elapsed time, or explicit missing proof> Correction: <profile split or rule change> Verification: <before/after commands and result> Residual risk: <what remains intentionally gated and where>
Gotchas
- A fresh evaluator is an independence control, not a release-signing check.
- A VM can be a reusable execution environment without forcing release identity checks into every VM smoke.
- A green fast gate does not prove release readiness; a red release-only gate does not invalidate a staging smoke unless the staging claim depends on it.
- Do not replace a misplaced gate with retries, sleeps, or a bypass marker.
- Do not infer overload from one slow run; distinguish environment failure from a profile contract error.
Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| Staging smoke asks for signing | Release gate leaked into staging profile | split release-attestation and run the smoke on the unsigned staging artifact |
| Security proof blocks on a production VM | Runtime environment and release identity are coupled | keep the VM, remove release-only assertions from the security profile |
| Same gate fails repeatedly | Wrong scope, flaky boundary, or missing fixture | classify the failure and add a focused reproducer; never silently retry |
| Agent says "tests passed" with no profile | Evidence contract is incomplete | require the report fields above and the exact command/result |
| Fix removes a safety check | Causal ownership was not traced | restore the check, document the narrower boundary, and re-verify it there |
