Release Sample Sweep

Release sample sweep (chrome-devtools)

Before a release, every user-facing surface is exercised the way a user exercises it. This skill is that procedure written down.

Three surfaces, three drivers — all three are release gates:

When to run it

• Before cutting any release. Non-negotiable — it is the last gate before release-4x.yml.
• After a change to the Console bundle (modules/spring-boot-starter/frontend/), since the Console is both the shipped sample UI and the validation surface.
• After a change to a shared module that every sample transitively depends on (modules/cpr, modules/ai, modules/spring-boot-starter, modules/admin).
• After a dependency bump wave — two of the last three sweeps found a version-skew bug that compiled clean and only failed at runtime.

What it catches that CI does not

CI builds and tests modules; this sweep exercises packaged artifacts in a browser. The gap between those is where the real bugs live:

scripts/release-gate-samples.sh automates the boot-and-assert half of this in CI. This sweep is the browser half on top of it — the layer that sees rendering, streaming, transport headers, tool cards, and console errors.

The shape of the sweep

Copy code
Step 0   Preconditions  — build everything, start Ollama, free the ports, open the ledger
Step 1a  Samples        — 33 samples: launch → drive → collect → verdict → teardown
Step 1b  Expo client    — the RN client in the iOS simulator
Step 1c  CLI            — atmosphere run/new/compose/import/checkpoint + distributions
                           ALL OF PHASE 1 IS COLLECT-ONLY. Do not fix anything mid-sweep.
Step 2   Triage         — classify every finding, rank by blast radius
Step 3   Fix            — root-cause fix + a regression test per issue, in the right
                           suite, each proven to bite
Step 4   Re-test        — the failed surfaces in full, plus the blast-radius subset
                           of already-passing ones
Step 5   Report         — vault report, CI green, memory updated

Step 1 is deliberately fix-free. Fixing mid-sweep changes the artifact under test and invalidates every sample already verified against the old one. The one exception: a defect that blocks the sweep itself from continuing — fix it, say so in the ledger, and note which already-passed samples were re-run.

Non-negotiables

1. chrome-devtools, never curl, for validation. curl is allowed only for port readiness and for headless wire protocols (A2A/MCP/REST) that serve no HTML. A "works via curl" claim skips the whole JS layer and is a false pass.
2. The Atmosphere Console is the UI. Drive /atmosphere/console/ (Spring Boot samples redirect / there). A sample that needs a bespoke page instead of the Console is itself a finding.
3. Assert the rendered element, not the payload. An image node with a src is a rendered screenshot; the same base64 in a StaticText node means nothing rendered it. "Server started", "HTTP 200", and "bytes present in the DOM" are not passes.
4. Boot the packaged artifact. java -jar (or quarkus-run.jar), never spring-boot:run / quarkus:dev. Both historical bugs above existed only at artifact level.
5. Kill by PID, never pkill -f. Never touch a port or process the sweep did not start — if a port is occupied, move to another port. The same rule covers the machine's network: never run networksetup, never take an interface down, never touch VPN/DNS/proxy settings. The host's Wi-Fi carries every session the maintainer has open, and a sweep interrupted mid-toggle can leave the machine offline indefinitely. Any assertion that needs real network loss is recorded PARTIAL with its unit coverage cited — see references/expo-sweep.md.
6. Model limitation ≠ framework bug. A small local model emitting invalid tool-call arguments is a model limitation; record it as such and prove it by re-running the same flow on a capable model before calling it a regression.
7. Never write "flaky". Reproduce it, or explain the mechanism. If neither is possible yet, it is a FAIL with an open question, not a dismissal.
8. Report honestly. PASS / PARTIAL / FAIL with one line of concrete evidence each. PARTIAL must name what was not proven and why.

Step 0 — Preconditions

bashCopy code
git status --porcelain                    # must be clean
git rev-parse --short HEAD                # record this SHA in the ledger
grep -m1 '<version>' pom.xml              # record the version under test

./mvnw install -DskipTests -Pfastinstall  # full reactor: framework + every sample jar
./scripts/sync-console-bundle.sh --check  # the Console you will drive must be current

ollama list                               # qwen2.5:3b + qwen2.5:7b-instruct-q4_K_M
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:11434/v1/models

• LLM backend is local Ollama, keyless. Use qwen2.5:3b for streaming samples and qwen2.5:7b-instruct-q4_K_M for tool-heavy agents — 3b emits invalid tool-call arguments and Ollama answers 400. Note real-ollama is a CI-harness alias only; AiConfig matches the literal local.

• The launcher scrubs ambient LLM env (LLM_API_KEY, LLM_BASE_URL, LLM_MODE, LLM_MODEL, and the provider keys) from every sample it boots, so the sweep is reproducible on any machine. SWEEP_KEEP_ENV=1 inherits instead. If you boot a sample by hand, scrub them yourself — a maintainer's profile routinely exports these.

• Always read the resolved endpoint out of the boot log before driving:

  Copy code
  grep 'AI config:' target/sweep/<sample>.log
  

  Expect mode=local … endpoint=http://localhost:11434/v1. Anything else means the sample is not talking to Ollama and the turn's result says nothing about this build. An explicit LLM_BASE_URL outranks the mode by design, so an inherited one silently redirects a "local" run to a remote provider — that is what happened on the 2026-08-07 shakedown before the scrub existed.

• Do not use a paid key. The paid-LLM lane is retired; quota starvation is what made the 2026-06 sweep report nine samples as plumbing-only.

• Do not use embacle (embacle-server --provider claude_code) for tool-calling samples — it applies the host CLI's own configuration to responses and its tool-call fidelity is inconsistent. It is only useful to demonstrate "a capable model completes this flow cleanly", then stop it.

• Ports: the sweep runs on the 9101+ block so it never collides with the samples' own defaults or the Playwright fixture's 8080–8104. Assignments are in references/sample-matrix.md.

• Open the ledger at claude_docs/sample-sweep-<YYYY-MM-DD>.md (a gitignored symlink into the vault, so it survives context compaction). Template: assets/ledger-template.md. Write each row as you finish that sample, never in a batch at the end.

Step 1a — The per-sample loop

Work through references/sample-matrix.md in order. For each sample:

bashCopy code
# 1. Launch (the helper refuses to boot if the port is already answering)
.claude/skills/release-sample-sweep/scripts/sweep-sample.sh start <sample> \
    --port <9101+n> --ready-path <path> --env LLM_MODE=local --env LLM_MODEL=qwen2.5:3b

2. Fresh browser page per sample — new_page on the drive URL. Never reuse the previous sample's page: stale state and leftover console noise both corrupt the evidence.
3. Snapshot — take_snapshot. Confirm the Console mounted and the transport badge reads what the matrix expects (Connected · websocket / · webtransport / · grpc / · ag-ui). A transport that silently fell back is a finding.
4. Drive the headline flow for that sample — the exact interaction is in the matrix, the mechanics per surface class are in references/driving-recipes.md.
5. Wait for the rendered result — wait_for the expected text/element, then re-take_snapshot and confirm the node type (see non-negotiable #3).
6. Collect the evidence, all three sources:
   • list_console_messages — every error and warning, verbatim
   • list_network_requests — any non-2xx/failed request
   • sweep-sample.sh warnings <sample> — server-side WARN/ERROR/exception/SLF4J Record warnings even when the sample passes. The warning inventory is half the value of the sweep and is what the next release's triage starts from.
7. Verdict + one-line evidence into the ledger:
   • PASS — headline feature observed rendered, no unexplained console error, no server exception.
   • PARTIAL — plumbing proven, headline feature not observed, reason named (missing third-party key, Docker unavailable, model limitation).
   • FAIL — feature broken, error frame, exception, or the sample won't boot.
8. Teardown — close_page, then sweep-sample.sh stop <sample>. The helper verifies the port is actually released; if it is not, stop and investigate before the next sample claims it.

Step 1b — The Expo client

samples/spring-boot-ai-classroom/expo-client/ is a native Expo/RN app. It is not a Maven module, not in cli/samples.json, and unreachable by the Playwright suites — this sweep is its only gate. It links atmosphere.js by file path, so it is also the only pre-release check that the client library's ./react-native export works in a real RN runtime.

Driven with the iOS simulator MCP, not chrome-devtools. Full procedure, including the SERVER_URL port trap and the AppState/NetInfo assertions nothing else covers: references/expo-sweep.md.

Step 1c — The CLI

The CLI is the documented Quick Start and ships as four distributions (curl installer, npx, Homebrew tap, SDKMAN). CI covers list/info, argument validation, the runtime overlays, and the installers — it never boots a sample through atmosphere run and looks at the UI, and it has no coverage for compose or checkpoint.

The manual pass closes that: atmosphere run → browser-driven, atmosphere new → scaffold + compile against Maven Central, plus the thin-coverage commands and a post-publish check of the actually-shipped artifacts. Watch the jar cache — a stale $ATMOSPHERE_HOME/cache/v<version> boots the previous release and fakes a pass. Full procedure: references/cli-sweep.md.

Step 2 — Triage

With all samples tested, classify each finding before touching any code:

Rank by blast radius: shared-module findings first (they can invalidate other samples' passes), then per-sample.

Step 3 — Fix, with a regression test per issue

Every issue gets a test, but in the suite that can actually run it:

For every issue in the framework-bug or sample-bug class:

1. Root-cause it first. Read the failing path; do not pattern-match a fix.
2. Smallest change that fixes the cause — the 2026-07 langchain4j fix was a single pom property.
3. Write a Playwright e2e spec that reproduces the failure, in the right home, wired into the right CI lane, and proven to bite: it must fail against the pre-fix artifact and pass after. Recording only "it passes now" proves nothing. Full authoring + wiring recipe: references/regression-specs.md.
4. Where a build-time lint can close the whole class, add that too — the langchain4j fix shipped both a spec and SampleLangChain4jVersionLintTest, which fails the build if any sample pom hardcodes a LangChain4j version.
5. Rebuild the affected modules and samples before re-testing.
6. One commit per fix class, conventional-commit prefixed. No CHANGELOG edits — the CHANGELOG is touched only at release time.

Step 4 — Re-test

1. Re-run the failed sample end to end — the full headline flow, not just the broken step.
2. Re-run the blast-radius subset of already-passing samples. The fix changed the artifact those passes were recorded against, so their evidence is only still valid if the fix could not reach them. The mapping from "what the fix touched" to "which passing samples must be re-driven" is in references/retest-subset.md.
3. Say what you did not re-run and why. A subset is a deliberate scope decision; leaving it unstated reads as "everything was re-verified".
4. Repeat triage → fix → re-test until the sweep is clean.

Step 5 — Report

• Vault report via the obsidian-writer skill → Claude Outputs/Sample-Sweep-chrome-devtools-<date>.md. Promote the ledger: full matrix with the evidence column, the issues-found-and-fixed section with commit hashes, methodology caveats, and non-blocking follow-ups.
• Every number verified — sample count from ls samples/ (minus shared-resources) and cli/samples.json, never from memory.
• CI green on the fix commits before the release proceeds — all workflows, not just the one you were watching.
• Update memory with anything reusable: a new trap, a new drive recipe, a changed port, a sample added or removed.

Files in this skill