Redamon Testing

When to Use

• Writing or fixing a test anywhere in the repo, or deciding where a new test goes.
• A test is red/skipped/xfailed and you must decide whether it is real.
• You were asked to run the suite or verify a change "works".

The repo-wide rule "never validate with host pytest, use the Docker gate" lives in the root AGENTS.md CRITICAL RULES; this skill is everything after that: isolation, tiers, and how to write a test that asserts something.

Critical Rules

• NEVER run pytest across a whole tree in one process. Many tests stub langchain/langgraph into sys.modules and bake tool objects against a fake @tool at import time, so whichever file collects first decides for all of them. You get phantom failures in files you never touched (classically a coroutine was expected, got <MagicMock>). Run ./redamon.sh test, or one file / node id. The gate exists for this: tooling/scripts/pytest_isolated.py runs each FILE in its own subprocess.
• NEVER "fix" source because a test went red in a multi-file run. Re-run that one file in isolation first; if it passes alone the failure was pollution, not a bug.
• NEVER print("SKIP..."); return to skip a test. pytest records that as PASSED while asserting nothing. Use self.skipTest(...) inside a TestCase or pytest.skip(...) in a bare function.
• NEVER read a green gate as "the live checks passed". A self-skipping test that needs a service prints SKIP and exits 0, and the gate containers have no Neo4j, so the live-graph schema checks in recon/tests/test_schema_catalog.py SKIP in every CI run. Their hermetic counterparts in recon/tests/test_graph_writes_documented.py do run. Neither sees everything: the code scan cannot see the ~22 labels written with SET n += $props (the names are built in Python and appear in no file), and the live graph cannot see a feature this deployment never ran. After a schema change, run the live one against a stack before believing it.
• NEVER rewrite an assertion so it passes. If a test reveals a real bug, mark it @pytest.mark.xfail(strict=True, reason=...) and say so. Tests must not enshrine bugs.
• NEVER put a recon test in the root tests/ folder. Root tests/ runs in the agent image; recon files there must be listed in _ROOT_RECON_TESTS at redamon.sh:4476 or they run against the wrong image and fail on imports. New recon tests go in recon/tests/.
• NEVER add a third-party import to a test without checking it is in the section image. Only pytest, pytest-cov, pytest-xdist, pytest-asyncio (requirements-test.txt) are guaranteed; anything else errors the whole file at collection. Prefer unittest.mock and the stdlib.
• ALWAYS assert behaviour, not execution. For a tool wrapper, assert both the parsed result and the command that was built. Verify the patch target against the source (recon/tests/test_arjun.py broke when subprocess.run became Popen and the mocks kept targeting run).
• ALWAYS make a test that needs a stack, binary, service or git HEAD skip cleanly. A hard failure on a missing prerequisite is a bug in the test.

Assert the command a wrapper BUILDS (the direction most often skipped)

pythonCopy code
from recon.helpers.nuclei_helpers import build_nuclei_command   # the seam under test

cmd = build_nuclei_command(targets_file="/tmp/t.txt", output_file="/tmp/o.jsonl",
                           docker_image="projectdiscovery/nuclei:latest", dast_mode=True)
assert "-dast" in cmd            # the flag we asked for is present
assert cmd.count("-dast") == 1   # and not duplicated by a second code path

Reference: recon/tests/test_nuclei_two_pass.py. For a wrapper that also parses tool output, mock the tool (patch subprocess.run) and assert both the parsed result and the command from mock_run.call_args.

Where a test goes, and its tier

Tier is auto-assigned by filename in each conftest.py (live checked first). An explicit @pytest.mark.{unit,integration,live} wins, on the file or on a single test - the gate passes the tier to pytest as -m, so an opted-out test inside a unit-named file really is skipped:

Commands

bashCopy code
./redamon.sh test                 # unit gate, every section + webapp vitest; must be 100% green
./redamon.sh test all             # unit + integration (NOT live)
./redamon.sh test coverage        # per-section floor via REDAMON_COV_FLOOR
./agentic/run_tests.sh            # agent section only; per-file isolated
./agentic/run_tests.sh tests/test_foo.py::TestX::test_y   # single node id (already isolated)

An unbuilt image is skipped, not failed - read the section headers before trusting "all green".

Resources

• docs/readmes/README.TESTING.md - full testing guide + coverage ratchet
• tooling/scripts/pytest_isolated.py - the per-file isolation gate
• redamon.sh:4479 - _TEST_SECTIONS, section/image map, shell + webapp hooks
• Related: root AGENTS.md for the host-pytest / Docker-gate rule