Ai Evaluation

AI Evaluation and Monitoring

Operating contract

Inputs

Outputs

• Produce: evaluation plan, scored results, confidence and failure slices, release verdict, and monitoring thresholds.

Capability and permission boundaries

Default to read-only analysis. Read only scoped records; redact secrets and regulated data. Writes, execution, network calls, production configuration, customer communication, billing changes, and delegation require explicit authority and an identified owner. Never widen tenant, time-window, or system scope implicitly.

Degraded mode

When required telemetry, evidence, execution, network access, or write authority is unavailable, return a partial result with each unassessed item labelled, preserve the safest existing state, and state the evidence or approval needed to continue. Never convert missing evidence into a pass.

Decision rules

Anti-Patterns

• Treating absent evidence as success. Fix: mark the check unassessed and name the missing source.
• Expanding one tenant or workflow to all tenants. Fix: enforce supplied scope at every query and action.
• Performing a production write during analysis. Fix: emit a reviewed change plan until authority is explicit.
• Reporting a metric without population, window, or source. Fix: attach all three.
• Hiding a failed threshold inside an average. Fix: report failure slices and the remediation owner.

Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.

Use When

• Use when setting up quality assurance for AI features — defining evaluation criteria, measuring output quality, using AI-as-judge, monitoring production AI, detecting drift, and building user feedback loops

Evidence Produced

References

• Use the links and companion skills already referenced in this file when deeper context is needed.
• Load ai-llm-integration/references/llm-application-pattern-decision-catalogue.md (section 4) when the task is the eval-driven development loop or reusing evals as runtime guardrails.

Overview

Evaluation is the biggest bottleneck to successful AI deployment. Define evaluation criteria BEFORE building. Without evaluation, you cannot know if your AI feature is working, degrading, or harming users.

Core principle: Evaluation-driven development. Like TDD for AI — define what "good" means first, then build.

Evaluation Contract

Every production AI feature needs an evaluation contract before release:

• Task definition: what the model must do, for whom, in which workflow.
• Business metric: time saved, revenue influenced, conversion lift, defect reduction, service quality, risk reduction, or cost avoided.
• Quality metrics: correctness, completeness, relevance, tone, actionability, citation quality, and domain compliance.
• System metrics: latency, cost per successful task, availability, retry rate, fallback rate, and tool failure rate.
• Safety metrics: PII exposure, prompt-injection resistance, harmful output, policy violations, and unauthorized action attempts.
• Release threshold: minimum pass rates and maximum regression allowed versus the current production baseline.
• Rollback trigger: metrics or incidents that require disabling the feature or reverting prompt/model/tool versions.

Evaluation Dimensions

Evaluation Workflow

Prompt and agent evaluation addendum

Version the prompt builder, model identifier, context assembly, tool policy, and evaluation data as one change surface. For each revision, record the objective, hard constraints, expected output, risk class, and acceptance grader. Compare against the previous version on representative, edge, adversarial, multilingual, and low-context cases; report per-slice failures, not only the aggregate score. Include a regression case for every production incident. A successful demo or syntax check is not release evidence for an AI feature.

For agentic workflows, evaluate the whole loop: context selection, plan quality, tool-argument validity, authorization/approval, side effects, recovery, final output, latency, cost, and audit trace. Keep deterministic format/schema/security checks separate from model-judged quality checks. If a grader, test set, runtime, or account model is unavailable, mark that result NOT ASSESSED rather than treating absence as a pass.

Copy code
1. Define criteria before building
2. Create golden test set (20–50 examples with expected outputs)
3. Run automated format checks on every new model/prompt version
4. Run AI-as-judge for quality checks
5. Compare against previous version — only deploy if metrics hold or improve
6. Monitor production: track live metrics + user feedback
7. Retrain/reprompt when drift detected

For features that recommend, generate, rank, automate, or trigger consequential actions, load references/ai-human-oversight-evaluation.md. Evaluate the problem, human, system, model, inputs, and outputs separately. Do not call a model-quality pass a product-quality pass. Require evidence for correction, contest, undo or safe fallback, escalation, consent/notice, and the audit event.

When a model, score, rule set, or ranking decides or materially shapes an outcome for a person (credit, hiring, eligibility, pricing, visibility, risk flags), load references/algorithmic-decision-impact-review.md for the impact tier, proxy and feedback-loop checks, subgroup error evidence, fairness-criterion ownership, and contest path required before release.

Evaluation Dataset Design

• Build separate sets for development, release gating, adversarial testing, and production monitoring.
• Include real examples, expert-created examples, edge cases, multilingual/local examples, low-resource context, and abuse attempts.
• Version each case with task, tenant/domain, source, expected behavior, rubric, and data sensitivity.
• Keep leakage out of evals: do not let the same generated examples drive both prompt tuning and final release scoring.
• Refresh cases after production incidents, new user behavior, platform changes, policy changes, or model migrations.

Release Gates

Do not ship an AI feature because a demo looked good. Ship it because it passes the evaluation contract.

Creating a Golden Test Set

sqlCopy code
CREATE TABLE ai_eval_cases (
  id              INT AUTO_INCREMENT PRIMARY KEY,
  feature_name    VARCHAR(100),       -- 'invoice_analysis', 'sales_report'
  input           TEXT NOT NULL,      -- the user query or document
  expected_output TEXT,               -- ideal output (or key elements of it)
  eval_criteria   JSON,               -- {"format": "json", "must_contain": ["total", "vendor"]}
  created_by      INT,
  created_at      TIMESTAMP DEFAULT NOW()
);

Build test cases from:

1. Real production queries (once live)
2. Domain expert-crafted examples
3. Edge cases: empty input, wrong language, very long input, adversarial input

Automated Evaluation (No LLM Cost)

Run these on every deployment:

phpCopy code
class AiEvaluator {
    public function evaluateFormat(string $output, array $criteria): EvalResult {
        $score = 0;
        $issues = [];

        // JSON validity
        if (($criteria['format'] ?? null) === 'json') {
            $decoded = json_decode($output, true);
            if (json_last_error() !== JSON_ERROR_NONE) {
                $issues[] = 'Invalid JSON';
            } else {
                $score += 25;
                // Required keys
                foreach ($criteria['required_keys'] ?? [] as $key) {
                    if (!array_key_exists($key, $decoded)) {
                        $issues[] = "Missing key: $key";
                    } else {
                        $score += 10;
                    }
                }
            }
        }

        // Length constraints
        if (isset($criteria['max_words'])) {
            $wordCount = str_word_count($output);
            if ($wordCount > $criteria['max_words']) {
                $issues[] = "Too long: {$wordCount} words (max {$criteria['max_words']})";
            } else {
                $score += 15;
            }
        }

        // Must-contain terms
        foreach ($criteria['must_contain'] ?? [] as $term) {
            if (stripos($output, $term) === false) {
                $issues[] = "Missing expected term: $term";
            } else {
                $score += 10;
            }
        }

        return new EvalResult($score, $issues);
    }
}

AI-as-Judge

Use a strong model to evaluate your AI feature's outputs. Reliable for quality, relevance, and tone.

phpCopy code
function judgeAiOutput(string $input, string $output, string $criteria): array {
    $judgePrompt = <<<PROMPT
You are evaluating the quality of an AI assistant's response.

Evaluation criteria:
{$criteria}

User input:
---
{$input}
---

AI response to evaluate:
---
{$output}
---

Score the response on each criterion from 1–5 (5 = excellent).
Explain your reasoning briefly, then give an overall score (1–5).
Format your response as JSON:
{
  "relevance": {"score": X, "reason": "..."},
  "accuracy": {"score": X, "reason": "..."},
  "tone": {"score": X, "reason": "..."},
  "overall": X
}
PROMPT;

    return callLLM('gpt-4o', $judgePrompt, temperature: 0.1);
}

AI-judge best practices:

• Use a stronger model as judge than the one being evaluated
• Ask for reasoning BEFORE score (reduces positional bias)
• Use pairwise comparison (A vs B) for relative quality rather than absolute scores
• Multiple judges + average for high-stakes decisions
• Watch for self-serving bias (GPT-4 favours GPT-4 outputs)

Production Monitoring

Metrics to Track Per Feature

sqlCopy code
CREATE TABLE ai_quality_metrics (
  id             BIGINT AUTO_INCREMENT PRIMARY KEY,
  tenant_id      INT NOT NULL,
  feature_name   VARCHAR(100),
  prompt_version VARCHAR(10),
  model          VARCHAR(50),
  format_valid   BOOLEAN,
  latency_ms     INT,
  tokens_in      INT,
  tokens_out     INT,
  judge_score    DECIMAL(3,2),         -- 1.00–5.00 from AI judge (async)
  user_rating    TINYINT,              -- 1–5 from explicit feedback
  thumbs_up      BOOLEAN,              -- quick user feedback
  created_at     TIMESTAMP DEFAULT NOW(),
  INDEX idx_feature_date (feature_name, created_at),
  INDEX idx_tenant_date (tenant_id, created_at)
);

Key Metrics by Priority

1. Format failure rate — % of responses failing JSON/schema validation
2. User thumbs-down rate — explicit negative feedback
3. Early termination rate — user stops generation mid-way
4. Average judge score — from async AI-as-judge on random sample
5. p50/p90/p99 latency — track at percentiles, not average
6. Cost per request — tokens × price per token

Alerting Thresholds

phpCopy code
$alerts = [
    'format_failure_rate' => 0.05,    // Alert if > 5% of responses fail format
    'thumbs_down_rate'    => 0.15,    // Alert if > 15% negative feedback
    'p99_latency_ms'      => 8000,    // Alert if p99 latency > 8 seconds
    'cost_per_request'    => 0.05,    // Alert if avg cost > $0.05 per request
];

Drift Detection

Drift = your AI feature is silently getting worse. Causes:

1. Model API updates — providers silently update model versions
2. System prompt edits — even small changes change behaviour
3. User behaviour shift — users learn to write differently over time
4. Data drift — RAG documents become stale

Detection

sqlCopy code
-- Weekly average quality score — watch for downward trend
SELECT
    YEARWEEK(created_at) AS week,
    feature_name,
    AVG(judge_score) AS avg_quality,
    AVG(CASE WHEN thumbs_up = FALSE THEN 1 ELSE 0 END) AS negative_rate,
    AVG(latency_ms) AS avg_latency
FROM ai_quality_metrics
WHERE tenant_id = ? AND created_at > DATE_SUB(NOW(), INTERVAL 8 WEEK)
GROUP BY week, feature_name
ORDER BY week;

Act when:

• Average quality score drops > 0.5 points vs last 4-week average
• Format failure rate doubles vs baseline
• User negative feedback rate increases > 5% week-over-week

User Feedback Signals

Collect user edits as preference data: original output = rejected, edited version = preferred.

Evaluation Before vs After Deployment

Never deploy a new prompt or model without running the golden test set first.

Anti-Patterns

• No golden test set — you cannot measure regression
• Only measuring average latency — track p90/p99; outliers hurt users
• Skipping evaluation to ship faster — silent quality degradation is worse than delay
• No prompt versioning — you cannot roll back a broken prompt
• Judge uses same model as evaluated — self-serving bias inflates scores
• No user feedback mechanism — your most valuable signal goes uncollected

Sources

Chip Huyen — AI Engineering (2025) Ch.3–4,10; Chip Huyen — Designing ML Systems (2022) Ch.8

Multi-Tenant Eval Harness

This skill covers evaluation concepts. The engineering of the eval harness as a control-plane service in a multi-tenant SaaS — per-tenant golden datasets, CI gate that blocks regressing prompt/model changes, judge-LLM calibration, production sampling, drift detection, and pre-promotion gates for flagship tenants — lives in ai-eval-harness.

Cross-references:

• ai-eval-harness — the harness as a service.
• ai-hallucination-slo-and-grounding — uses eval signals to budget hallucination.
• ai-feature-rollout-and-experimentation — gates rollouts on eval.
• ai-on-saas-architecture — control-plane positioning.

Consolidated Child References

• Load references/routing.md to map retired AI child skill slugs to their reference modules.