Release Openclaw Plugin Testing

OpenClaw Pre-Release Plugin Testing

Use this skill when the user asks for plugin release confidence, plugin lifecycle sweeps, package-artifact plugin proof, or "what else should we test before release?" It complements openclaw-testing; use that skill too when choosing the cheapest safe runner or debugging a failing lane.

Goal

Prove the plugin system as a product surface, not just as source tests:

• bundled plugin lifecycle: install, inspect, enable, disable, uninstall
• package artifact behavior from a clean HOME
• doctor/fix/config validation and idempotence
• config discovery and config round-trip
• status/log visibility and diagnostics
• gateway startup/bootstrap with plugin metadata snapshots
• public SDK compatibility for real external plugins
• live-ish provider/channel probes only when safe credentials exist

First Checks

From the OpenClaw repo root:

bashCopy code
pnpm docs:list
git status --short --branch
pnpm changed:lanes --json

Follow openclaw-testing for dependency ownership and the choice of local or remote proof.

Runner Choice

Prefer this order:

1. GitHub Package Acceptance for installable-package product proof.
2. Current dedicated Linux worker for trusted source/package/Docker proof when it has the required dependencies and capabilities.
3. ci-build-artifacts-testbox.yml Testbox when Docker/package lanes need seeded dist, dist-runtime, and package caches.
4. ci-check-testbox.yml Testbox for source checks, targeted Vitest, package-boundary checks, or focused Docker lanes.
5. Workstation targeted commands only for small format/static/unit probes.

Avoid long package Docker runs from a stale sparse worktree. If Testbox sync reports hundreds of changed files or starts deleting package inputs, stop and warm a fresh box from current main, or switch to Package Acceptance.

Existing Baseline

Run or verify these before inventing new coverage:

bashCopy code
OPENCLAW_TESTBOX=1 pnpm check:changed
pnpm run test:extensions:package-boundary:canary
pnpm run test:extensions:package-boundary:compile
pnpm test:docker:plugins
OPENCLAW_PLUGINS_E2E_CLAWHUB=0 pnpm test:docker:plugins
pnpm test:docker:plugin-update

For full bundled install/uninstall proof, shard the packaged sweep:

bashCopy code
OPENCLAW_BUNDLED_PLUGIN_SWEEP_TOTAL=8 \
OPENCLAW_BUNDLED_PLUGIN_SWEEP_INDEX=<0-7> \
pnpm test:docker:bundled-plugin-install-uninstall

This example partitions the selected package's plugin inventory over shards 0-7. Private QA plugins are source-mode only unless a package explicitly includes them.

Confidence Matrix

Use this matrix for pre-release signoff. Record pass/fail, run URL/Testbox ID, package SHA/version, and skipped-live reason.

Package Acceptance Plan

Use this when validating a release branch, beta, or candidate package:

bashCopy code
gh workflow run package-acceptance.yml \
  --repo openclaw/openclaw \
  --ref main \
  -f workflow_ref=main \
  -f source=ref \
  -f package_ref=<branch-or-sha> \
  -f suite_profile=custom \
  -f docker_lanes='plugins-offline plugin-update doctor-switch update-channel-switch config-reload mcp-channels npm-onboard-channel-agent' \
  -f telegram_mode=mock-openai

Use source=npm -f package_spec=openclaw@beta for published beta proof. Keep workflow_ref as trusted current harness code unless the release process says otherwise.

For extended-stable shared publication, require complete exact-target Full Release Validation from the trusted main-pinned release-ci/* harness. Direct canonical-branch or main producers do not satisfy the protected publisher. Package Acceptance is a post-publish selector smoke:

bashCopy code
gh workflow run package-acceptance.yml \
  --repo openclaw/openclaw \
  --ref main \
  -f workflow_ref=main \
  -f source=npm \
  -f package_spec=openclaw@extended-stable \
  -f suite_profile=package \
  -f telegram_mode=mock-openai

Record the resolved version. Still verify every package and selector in the tag's all-publishable inventory; one smoke is not registry readback.

Plugin npm Artifact Preflight

Use the trusted main workflow to prepare and read back a selected plugin npm artifact from an exact release SHA without entering any publish approval, environment, secret, OIDC, npm mutation, or ClawHub mutation path:

bashCopy code
release_sha="$(git rev-parse origin/release/2026.7.1)"
gh workflow run plugin-npm-release.yml \
  --repo openclaw/openclaw \
  --ref main \
  -f preflight_only=true \
  -f publish_scope=selected \
  -f plugins=@openclaw/meta-provider \
  -f ref="${release_sha}" \
  -f npm_dist_tag=default

Do not pass release_publish_run_id. Require the workflow to finish verify_plugin_npm_preflight successfully. Record the run URL, workflow SHA, and source SHA. The workflow first creates the staging/readback artifact plugin-npm-package-source-<source-sha>-<extension-id> containing npm-pack.json, preflight-manifest.json, and the tarball. It then uploads the final consumer artifact plugin-npm-package-<extension-id>-<version>-<route>-<run-id>-<attempt> containing the tarball and plugin-publication-manifest.json.

Record the final artifact name and digest separately. The manifest uses openclaw.plugin-publication-artifact/v1 and records the target SHA, package manifest hashes, publication route and policy, and tarball hashes and inventory. This proof is validation-only; it does not authorize or stage publication. The separate trusted_publisher_preflight=true OIDC check requires a protected release-publish/<tooling-sha12>-<epoch> dispatch tag and runs in npm-publish. Real publication also requires that tooling tag; a direct human dispatch waits for its npm-release approval job before publishing. For an already-published version, require npm dist.integrity and dist.shasum to match the verified tarball. Treat only missing or provably older dist-tags as repairable; newer or incomparable selectors are a blocker.

New Testbox Harness Plan

If more certainty is needed, add or run a plugin-lifecycle-matrix Docker lane that uses one package tarball and sharded plugin lists. Per plugin:

1. Start with a clean HOME.
2. Capture plugins list --json.
3. plugins install <id>.
4. plugins inspect <id> --json.
5. plugins disable <id>, then assert disabled visibility.
6. plugins enable <id>, except config-required plugins without config.
7. plugins registry --refresh.
8. doctor --non-interactive.
9. plugins uninstall <id> --force.
10. Assert the plugin's plugins.entries value is exactly { enabled: false }, while its allow/deny entries, install record, managed directory, and bundled runtime load paths are gone. Use the existing harness's source-qualified uninstall expectations for historical targets.
11. Assert diagnostics contain no level: "error" and output redacts secret-looking values.

Keep memory-lancedb special: it is config-required. First assert install does not enable it without embedding config, then run a second configured case.

Doctor/Fix Matrix

Seed bad states and require doctor --fix --non-interactive to repair them, then run doctor again and require idempotence:

• stale plugins.allow
• stale plugins.entries
• stale channel config for missing channel plugin
• invalid plugins.entries.<id>.config
• packaged bundled path in plugins.load.paths
• legacy plugins.installs
• disabled channel/plugin config that must not stage runtime deps
• root-owned global package tree that must remain unmodified

Gateway Bootstrap Matrix

Start packaged OpenClaw in Docker with clean state:

• provider plugins enabled, no credentials: ready with warnings, no crash
• channel plugins configured disabled: no runtime deps staged
• startup-activation plugins enabled: ready and reflected in status
• invalid single plugin config: bad plugin skipped/quarantined, others remain

Assert:

• gateway reaches ready
• openclaw status --json includes plugin diagnostics
• openclaw plugins inspect --all --json is parseable
• package tree is not mutated
• logs contain no raw tokens

Config Round-Trip Representatives

Use representative plugin families instead of every plugin for deep config round-trip:

• providers: openai, anthropic, mistral, openrouter
• channels: telegram, discord, slack, whatsapp
• memory: memory-lancedb
• feature/runtime: browser, acpx, tokenjuice

For each representative:

1. Write config through CLI when possible.
2. Read it back through config get or JSON.
3. Run plugins inspect.
4. Run doctor --non-interactive.
5. Trigger gateway config reload if applicable.
6. Compare config hash before/after no-op commands.

External SDK Smoke

In a package Docker lane, create tiny external plugins and install them from:

• local directory
• .tgz
• file: npm spec

Cover CJS and ESM shapes, plus at least one plugin importing focused openclaw/plugin-sdk/* subpaths. Assert plugins inspect sees its tool, gateway method, CLI command, or service.

Live-Ish Probe Rules

Before live-ish work, source allowed env in Testbox and generate a redacted availability matrix: present/missing only, never values.

Only run probes for credentials that exist. Prefer auth/catalog/status probes over sending user-visible messages. If a probe might contact an external user, channel, or workspace, stop and ask the user.

Reporting

Report in this shape:

textCopy code
package/ref:
tbx ids / run urls:
matrix:
  bundled lifecycle:
  package acceptance:
  doctor/fix:
  gateway bootstrap:
  config round-trip:
  sdk external:
  live-ish:
failures:
skips:
next highest-value gap:

Say clearly when a failure is Testbox sync/env damage rather than product behavior, and prove that with a clean rerun or current-main comparison.