Test Engine

Skill: Power Apps Test Engine + Playwright

When to Use

Trigger when testing Power Platform apps end-to-end, writing automated test plans, setting up test pipelines, or verifying UI after deployment.

Decision Matrix: Which Testing Approach?

Rule of thumb: Start with Test Engine Power Fx for standard apps. Drop to raw Playwright only when Test Engine abstractions can't reach what you need (custom controls, third-party embedded UIs, visual regression).

AI Accuracy Warning

LLMs are less accurate with Power Fx than mainstream languages (JavaScript, C#, Python) due to smaller training datasets. Always verify Power Fx test steps manually before trusting them in CI/CD. This applies to both test authoring and formula column definitions.

⚠️ REQUIRED: Load Sub-Files Before Implementation

SKILL.md is a summary only — it is NOT sufficient for implementation.

The detailed content (complete payloads, XML templates, working examples, edge-case handling) lives in sub-files in the same directory as this SKILL.md. Before writing any code, you MUST use read_file on the sub-files relevant to your task:

• Test Plans (YAML Format) — YAML schema, Power Fx test steps, assertions, credentials, recording video
• Canvas App Testing — Canvas provider, control name selectors, gallery navigation, delegation testing
• Model-Driven App Testing — MDA provider, form verification, view checks, BPF testing, Dataverse integration
• Playwright Advanced — Direct Playwright functions, custom C# scripts, raw TypeScript tests, Sean Astrakhan's AI-generated test pattern
• CI/CD Integration — ADO pipeline YAML, GitHub Actions, publishing .trx results, regression testing on every deploy
• Power Pages Testing — Portal provider, auth testing, table permission verification, Web API testing

Providers Overview

Test Engine uses a provider-based architecture. Each provider is optimized for a specific application type:

Quick Start

powershellCopy code
# 1. Install prerequisites
# PAC CLI: https://learn.microsoft.com/power-platform/developer/cli/introduction
# Azure CLI: https://learn.microsoft.com/cli/azure/install-azure-cli

# 2. Authenticate
az login --allow-no-subscriptions
pac auth create --name TestAuth --url "https://contoso.crm.dynamics.com"

# 3. Set test user credentials (env vars — never in YAML)
$env:user1Email = "testuser@contoso.onmicrosoft.com"

# 4. Run a canvas app test
pac test run `
  --provider canvas `
  --test-plan-file "tests/smoke-test.te.yaml" `
  --tenant "your-tenant-id" `
  --environment-id "your-env-id"

# 5. View results
# Output: .trx file + screenshots/videos in output directory

Anti-Patterns

• Using raw Playwright when Test Engine control-name selectors would work (fragile DOM selectors break on platform updates)
• Storing credentials in test plan YAML files (use environment variables)
• Skipping test setup/teardown (test data left behind corrupts other tests)
• Running tests against production data without isolation
• Not recording video on CI/CD runs (impossible to debug failures without visual evidence)
• Trusting AI-generated Power Fx test steps without manual verification
• Using headless: true for initial test development (use headless: false to see what's happening)

Key References

• Test Engine Overview
• YAML Format
• Playwright Integration
• ALM Integration
• GitHub Repo
• Power Fx Functions