Revyl CLI Failure Analysis Skill
Native Agent Behavior
- Ask at most 1-3 concise clarification questions only when the target test, workflow, session, report URL, or sensitive action cannot be inferred from the repo or Revyl CLI.
- Prefer safe defaults and keep moving when
revyl test list,revyl workflow list,revyl test report,revyl workflow report,revyl device report, screenshots, or reports can answer the question. - When Revyl prints a report, viewer, or local app URL, open it in the native browser/tool surface when available: Codex Browser/in-app browser for local URLs, Revyl report/viewer URLs, screenshots, and page checks; Claude Code
.claude/skillscompatibility links plus WebFetch/WebSearch or configured MCP/browser tools; Cursor.cursor/skillswhen using--copy, otherwise shared.agents/skills, plus available MCP/browser tools. Codex also discovers shared.agents/skillsdirectly. - If no browser tool is exposed, report the URL and verify through
revyl test report,revyl workflow report, orrevyl device reportinstead of claiming browser access. - Confirm before entering sensitive data, submitting forms, uploading files, accepting browser permissions, changing sharing/access, or deleting data.
Quick Start
bash# 1) Pull structured evidence for test runs revyl test report <test-name> --json # Or pull structured evidence for a live/manual device session revyl device report --session-id <session-id> --json # 2) Classify failure # REAL BUG | FLAKY TEST | INFRA ISSUE | SETUP ISSUE | TEST IMPROVEMENT # 3) Apply fix and rerun revyl test run <test-name>
For workflow-level triage:
bashrevyl workflow report <workflow-name> revyl workflow report <workflow-name> --json
Decision Matrix
| Signal | Classification | Action |
|---|---|---|
| Instructions succeed but final state contradicts expected behavior | REAL BUG | File defect with expected vs actual evidence |
| App behavior acceptable but assertion wording too brittle | FLAKY TEST | Rewrite validation wording |
| No steps executed or setup failed | INFRA ISSUE | Re-run and inspect environment/device/build setup |
| Session remains on login, permission, onboarding, or recovery UI before the target flow | SETUP ISSUE | Complete or fix setup before feature testing |
| Test structure allows false positives or combines verify+action | TEST IMPROVEMENT | Restructure YAML |
Device Session Analysis
When a session-id is available, analyze that report directly. Do not only point at a previous successful session.
For setup-dependent sessions, explain the shape of what happened:
- Startup: platform, status, install/launch actions, dev-client deep link, and whether the relay or first bundle looked healthy.
- Setup entry: whether the session reached the expected starting screen and handled permission dialogs if they appeared.
- Credential handling: whether email/password were entered, but do not copy
type_data.value, raw credentials, signed URLs, or artifact URLs from the report. - Source of secrets: use test variables or
{{global.name}}placeholders in YAML; do not hard-code or repeat secrets. - Final state: the screen reached after setup and whether it proves the target flow is ready.
- Next action: if still on login/onboarding/recovery/permission UI, classify as
SETUP ISSUEand complete setup before feature testing.
Output Format
textTest/Session: <name or session-id> Result: <PASS/FAIL> Failure Step: <order> - <description> Classification: <REAL BUG | FLAKY TEST | INFRA ISSUE | SETUP ISSUE | TEST IMPROVEMENT> Confidence: <HIGH | MEDIUM | LOW> Session shape: - Startup: <platform/status/install-launch/deep-link readiness summary> - Setup path: <entrypoint, permission dialog, credential or setup flow> - Secret handling: <globals/placeholders used; no raw credentials copied> - Final state: <target-ready screen or remaining setup blocker> Evidence: - Expected: <description> - Observed: <reasoning summary> - Why this classification: <short rationale> Exact next action: - <bug report details OR yaml rewrite OR infra rerun command> Rerun command: - revyl test run <test-name>
