TypingMindTypingMindThe best frontend for AI
Analyze Evals logo

Analyze Evals

Organization
microsoft
analyze-evals

Analyze exported evaluation results from Copilot Studio's Evaluate tab. The user provides a CSV file exported from the Copilot Studio UI; this skill parses it, identifies failures, and proposes YAML fixes. No API access or published agent required — just the exported CSV.

Overview

Publishermicrosoft
Repositoryskills-for-copilot-studio
Skill nameanalyze-evals
Stars
478
Forks
101
Bundled files
Instructions only
LicenseMIT
Links
Skill sourceRepository
  • Markdown instructions

    A SKILL.md file the model loads on demand, so it only costs tokens when a request actually matches.

  • Works with any LLM

    AI skills are plain Markdown, not provider-specific code, so this works with GPT, Claude, Gemini, Grok, or a local model.

  • Self-contained

    Everything the model needs lives in the instructions — no extra files to sync.

  • Open source

    Published by microsoft on GitHub. Read the source before you install it.

Installation

Install the Analyze Evals AI skill in TypingMind to use it with any LLM, or drop it into another agent that reads SKILL.md.

1

Install in TypingMind

TypingMind installs a skill straight from its GitHub folder — it reads SKILL.md, bundles the resource files, and stores the result locally.

  1. Open the app and go to Plugins → Skills.
  2. Choose "Install from GitHub".
  3. Paste the skill folder URL below and confirm.
    https://github.com/microsoft/skills-for-copilot-studio/tree/main/skills/analyze-evals
  4. Enable the skill in any chat where you want it available.
Plugins → Skills → Add skill → From GitHub URL, then paste the folder URL and press Continue.
2

Install in another agent

Any agent that reads the Agent Skills format can use this skill — copy the folder into that agent's skills directory.

Claude Code — .claude/skills
git clone --depth 1 https://github.com/microsoft/skills-for-copilot-studio.git /tmp/skills-for-copilot-studio
mkdir -p .claude/skills
cp -r /tmp/skills-for-copilot-studio/skills/analyze-evals .claude/skills/analyze-evals
Restart Claude Code after copying so it picks up the new skill.

Use it in TypingMind

Enable Analyze Evals in any TypingMind chat and the model takes it from there. Its name and description sit in the system prompt, and the moment a request matches, the model loads the full instructions itself — you never invoke it by hand, and it costs no tokens until it is actually used.

The model loads Analyze Evals on its own as soon as a request matches it.

Works with any AI model

AI skills are plain Markdown instructions rather than provider-specific code, so Analyze Evals is not tied to the model it was written for. Install it once in TypingMind and use it with GPT-5, Claude, Gemini, Grok, DeepSeek, Mistral, Llama, or a local model you run yourself — all on your own API keys.

  • Loaded only when it is needed

    The system prompt carries just the name and description. The instructions are fetched on the first matching request, so an idle skill costs nothing.

  • Switch models mid-chat

    Because the skill is instructions rather than code, changing model does not break it — the next model reads the same SKILL.md.

Skill instructions

This is the SKILL.md content the model loads. Read it before installing — a skill is instructions your model will follow.

Analyze Copilot Studio Evaluation Results

Analyze evaluation results exported from the Copilot Studio UI as CSV.

Phase 1: Get Results

  1. Ask the user for the CSV file path if not already provided. The file is typically exported from Copilot Studio's Evaluate tab and named Evaluate <agent name> <date>.csv in their Downloads folder.

  2. Read the CSV file. The in-product evaluation CSV has these columns:

    ColumnMeaning
    questionThe test utterance
    expectedResponseExpected response (may be empty)
    actualResponseWhat the agent responded
    testMethodType_1Eval method (e.g., GeneralQuality)
    result_1Pass or Fail
    passingScore_1Score threshold (may be empty)
    explanation_1Why it passed/failed (e.g., "Seems relevant; Seems incomplete; Knowledge sources not cited")

    The _1 suffix indicates the first eval method. There may be additional methods (_2, _3, etc.) with the same column pattern.

Phase 2: Analyze Results

  1. Focus on failed evaluations (result_1 = Fail, or any result_N = Fail).

  2. For each failure, use the explanation column to understand the issue:

    • "Question not answered" — The agent couldn't handle the question. Check if there's a matching topic or knowledge source.
    • "Knowledge sources not cited" — The agent responded but didn't cite sources. Check knowledge source configuration and SearchAndSummarizeContent nodes.
    • "Seems incomplete" — The response was partial. Check topic flow for early exits, missing branches, or incomplete SendActivity messages.
    • Error messages in actualResponse (e.g., GenAIToolPlannerRateLimitReached) — These are runtime errors, not authoring issues. Flag them to the user as transient failures to retry.

Phase 3: Propose Fixes

  1. For each failure, identify the relevant YAML file(s):

    • Auto-discover the agent: Glob: **/agent.mcs.yml
    • Find the relevant topic by matching the test utterance against trigger phrases and model descriptions
    • Read the topic file to understand the current flow

  2. Propose specific YAML changes to fix each failure. Present them to the user as a summary:

    • Which test(s) failed and why
    • Which file(s) need changes
    • What the proposed change is (show the diff)

  3. Wait for user decision. The user can:

    • Accept all — apply all proposed changes
    • Accept partially — apply only some changes (ask which ones)
    • Reject — discard proposed changes and discuss alternative approaches

  4. Apply accepted changes using the Edit tool. After applying, remind the user to push and publish again before re-running evaluations.

Frequently asked questions

What does the Analyze Evals AI skill do?

Analyze exported evaluation results from Copilot Studio's Evaluate tab. The user provides a CSV file exported from the Copilot Studio UI; this skill parses it, identifies failures, and proposes YAML fixes. No API access or published agent required — just the exported CSV.

Why use Analyze Evals on TypingMind?

Because you install it once and use it with any model. Analyze Evals is plain Markdown rather than provider-specific code, so the same skill runs on GPT-5, Claude, Gemini, Grok, or a local model — and you can switch model mid-chat without it breaking. TypingMind runs on your own API keys, so you pay providers directly instead of a per-seat subscription, and your skills and chats stay in your own storage.

How do I install Analyze Evals in TypingMind?

Open Plugins → Skills → Install from GitHub in TypingMind and paste https://github.com/microsoft/skills-for-copilot-studio/tree/main/skills/analyze-evals. TypingMind reads its SKILL.md and installs it as a skill you can enable per chat.

Which AI models can use Analyze Evals?

Any model you connect in TypingMind. AI skills are plain Markdown instructions rather than provider-specific code, so GPT, Claude, Gemini, Grok, and local models can all load this skill when a request matches it.

How many AI models can I use with Analyze Evals?

As many as you like. As long as a model supports skills, you can use Analyze Evals with it — GPT, Claude, Gemini, Grok, DeepSeek, Mistral, Llama and more — all on TypingMind with your own API keys.

Is the Analyze Evals AI skill free?

Yes. It is published on GitHub by microsoft under the MIT license. You only pay your own AI provider for the tokens you use.

What are AI skills?

An AI skill is a reusable instruction bundle that teaches an AI model how to do one specific task. It follows the open Agent Skills format: a SKILL.md file with a name and description, plus any scripts, templates or reference files the model may need. The model reads the instructions only when your request matches the skill, so an installed skill costs nothing until it is used.

How are AI skills different from plugins or MCP servers?

A plugin or MCP server gives a model new tools to call — code that runs somewhere and returns a result. An AI skill gives the model knowledge and process instead: how to approach a task, which steps to follow, what good output looks like. Skills are plain Markdown, so they need no server, no API key and no runtime, and they work with any model.

Related AI Skills

View all
mattpocock logo

Domain Modeling

mattpocock
CommunityPopular

Build and sharpen a project's domain model. Use when discussing codebase terminology, writing or editing a CONTEXT.md, or recording or editing an ADR.

22.3K
264.4K
3 files
View
mattpocock logo

Writing Fragments

mattpocock
CommunityPopular

Writing, explore: mine raw fragments, no structure yet.

22.3K
264.4K
1 files
View
mattpocock logo

Writing For Agents

mattpocock
CommunityPopular

Writing documents for agents. Use when creating or editing skills, or modifying AGENTS.md or CLAUDE.md.

22.3K
264.4K
2 files
View
NousResearch logo

Auteur

NousResearch
OrganizationPopular

Design and build cinematic, award-level web pages.

53.5K
250.4K
25 files
View

Set up your own AI workspace now

Get notified about new features and future giveaways by subscribing to our newsletter 👇