Genai Hub Prompt Evaluation

Step 0 — ask for credentials first (required)

Before running any other step, confirm the AI Core credentials are set. Invoke aicore-admin-resources and run check_setup.py.

• If it succeeds, continue to the workflow.
• If it fails (missing variables, invalid_client, or bad credentials), STOP and ask the user to export the variables listed in Environment variables below. Do not continue, and do not look for credentials or workflow steps anywhere else (notebooks, other repos, workspace files) — this skill's scripts/ are the only supported path.

Rules

1. Run all scripts from this skill's root directory with uv run scripts/<name>.py.
2. Shared lifecycle/admin steps are handled by other skills — invoke them rather than reimplementing: object stores and resource groups → aicore-admin-resources; artifact registration, executions, execution status → aicore-lifecycle-management.
3. Never print or log credential values — scripts read them from the environment via the SDK.
4. The evaluation scenario ID is genai-evaluations. Always use the executable ID genai-evaluations-simplified — hardcode it, do not use any version/user-suffixed variant. A tenant may list many such custom-built images; ignore them. Only fall back to discovery (aicore-lifecycle-management get_executables.py --scenario-id genai-evaluations) if genai-evaluations-simplified is not present on the tenant at all.
5. Before creating a custom metric or prompt-registry config, list existing ones and reuse by name/scenario/version to avoid duplicates. create_evaluation_metric.py already does create-or-get.
6. Only fetch results (get_evaluation_results.py) once the execution status is COMPLETED (UNKNOWN → PENDING → RUNNING → COMPLETED). Check status by invoking aicore-lifecycle-management (get_executions.py --execution-id <id>).
7. If a dataset has no reference/ground-truth column, do not select metrics that require a reference.

Environment variables

All scripts in this skill read credentials from the environment — never from CLI arguments.

AI Core credentials (required for all scripts)

Source: BTP cockpit → your subaccount → Instances & Subscriptions → AI Core instance → Service Keys.

Set them once via aicore configure (saves to ~/.aicore/config.json) or export each variable:

bashCopy code
export AICORE_AUTH_URL=https://<subdomain>.authentication.<region>.hana.ondemand.com/oauth/token
export AICORE_CLIENT_ID=<client-id>
export AICORE_CLIENT_SECRET=<client-secret>
export AICORE_BASE_URL=https://api.ai.<region>.aws.ml.hana.ondemand.com
export AICORE_RESOURCE_GROUP=default

Shell syntax: AICORE_RESOURCE_GROUP='default' — no space before =.

Object-store credentials (required only for download_evaluation_results.py)

The per-row SQLite DB is downloaded directly from S3 — AI Core credentials are not used here. Set the standard AWS SDK environment variables; the script uses boto3's credential chain directly:

The bucket name is passed as --bucket <name> on the command line (not read from env). Source: the same AWS credentials used when creating the object store secret in aicore-admin-resources.

End-to-end workflow

The full runbook (with every command and how the pieces connect) is in references/WORKFLOW.md. The high-level flow, and which skill owns each step:

Evaluation types

Choose one — it determines what you create in step 6 and which binding you pass to create_evaluation_configuration.py in step 7:

All three share steps 1–5 and 7–11; only step 6 and the config binding differ. The grounding type additionally needs grounding prerequisites (grounding-enabled resource group, grounding secret, document-grounding pipeline) — see references/GROUNDING_SETUP.md.

Evaluation metrics

bashCopy code
# Discover system metric names (use these verbatim in --metrics later)
uv run scripts/list_evaluation_metrics.py --kind system
uv run scripts/list_evaluation_metrics.py --kind system --name "Pointwise"

# List your custom metrics and their IDs
uv run scripts/list_evaluation_metrics.py --kind custom --json | jq '.[] | {id, name}'

# Create (or reuse) custom LLM-as-a-judge metrics from a definition file (idempotent).
# See templates/custom-metric.json. Prints the resulting metric ID(s).
uv run scripts/create_evaluation_metric.py --file templates/custom-metric.json

Common system metrics: Pointwise Conciseness, Pointwise Answer Relevance, Pointwise Instruction Following, Pointwise Correctness, Pointwise RAG Context Relevance, Pointwise RAG Groundedness, Pointwise RAG Context Precision, Pointwise RAG Completeness. Confirm the exact names for the tenant with list_evaluation_metrics.py --kind system.

Step 6 — create the thing to evaluate

Type 1 — Prompt registry

Create (or reuse) a prompt template in the Prompt Registry, then benchmark it across models.

bashCopy code
uv run scripts/list_prompt_templates.py --scenario genai-evaluations          # reuse?
uv run scripts/create_prompt_template.py --file templates/prompt-template.json \
  --name prompt-registry-eval-demo --version 1.0.0

Returns a prompt template ID → pass as --prompt-template-id, together with --models (e.g. "gpt-4o:2024-05-13") in step 7.

Type 2 — Orchestration registry

Create (or reuse) an orchestration config (prompt_templating, no grounding).

bashCopy code
uv run scripts/list_orchestration_registry_configs.py --scenario genai-evaluations   # reuse?
uv run scripts/create_orchestration_registry_config.py \
  --file templates/orchestration-registry-config.json \
  --name genai-eval-prompt-config --version 0.0.1

Returns a config ID → pass as --orchestration-registry-ids in step 7.

Type 3 — Grounding orchestration registry (RAG)

First complete the grounding prerequisites in references/GROUNDING_SETUP.md, then create the config from the grounding template (adds a grounding module):

bashCopy code
uv run scripts/create_orchestration_registry_config.py \
  --file templates/orchestration-registry-grounding-config.json \
  --name genai-eval-grounding-config --version 0.0.1

Returns a config ID → pass as --orchestration-registry-ids in step 7. Select RAG metrics (Pointwise RAG Groundedness, Pointwise RAG Context Relevance, …) for this type.

Step 7 — evaluation configuration + execution

Use the executable ID genai-evaluations-simplified directly (per Rule 4 — do not use a version/user-suffixed variant). Then create the configuration with this skill's script:

bashCopy code
# 2a. Create the evaluation configuration — ORCHESTRATION / GROUNDING types
uv run scripts/create_evaluation_configuration.py \
  --executable-id genai-evaluations-simplified \
  --dataset-artifact-id <artifact-id> \
  --test-dataset-path testdata/full_dataset.csv \
  --test-dataset-type csv \
  --metrics "Pointwise Correctness,Pointwise Answer Relevance" \
  --orchestration-registry-ids <registry-config-id> \
  --orchestration-deployment-url <orchestration-deployment-url> \
  --repetitions 1

# 2b. Create the evaluation configuration — PROMPT REGISTRY type
uv run scripts/create_evaluation_configuration.py \
  --executable-id genai-evaluations-simplified \
  --dataset-artifact-id <artifact-id> \
  --test-dataset-path testdata/full_dataset.csv \
  --metrics "Pointwise Correctness" \
  --prompt-template-id <prompt-template-id> \
  --models "gpt-4o:2024-05-13" \
  --orchestration-deployment-url <orchestration-deployment-url>

Custom metric IDs (from create_evaluation_metric.py) can be appended to --metrics. Limit rows with --test-row-count N. Map variables to dataset columns with --variable-mapping '{"all_metrics/reference":"data/ground_truth"}'.

Then hand off to aicore-lifecycle-management for the run itself:

• Start the execution — create_execution.py --configuration-id <config-id>
• Poll until COMPLETED — get_executions.py --execution-id <execution-id> (status goes UNKNOWN → PENDING → RUNNING → COMPLETED)

Results

bashCopy code
# Aggregate metrics per model/prompt run (only once COMPLETED)
uv run scripts/get_evaluation_results.py --execution-id <execution-id>
uv run scripts/get_evaluation_results.py --execution-id <execution-id> --json

For instance-level results (per-row scores, latency, logs), download the SQLite DB (<execution-id>/tmp/results.db) from object storage. Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment, then pass the bucket name via --bucket:

bashCopy code
uv run scripts/download_evaluation_results.py --execution-id <execution-id> --bucket <bucket> --show-tables

For the full download + local inspection recipe (table schema, querying per-row metrics), see references/RESULTS_ANALYSIS.md.

Handoffs

• Credentials, resource group, or object store secret not set up? → invoke aicore-admin-resources
• Need to start/stop/inspect executions, discover executables, or manage deployments? → invoke aicore-lifecycle-management
• Need a running orchestration deployment or want to design the prompt/RAG pipeline? → invoke genai-hub-orchestration
• Setting up grounding (RAG) for the grounding type? → see references/GROUNDING_SETUP.md (uses aicore-admin-resources for the resource group and secret)
• Evaluating an agent (not a prompt/LLM) via the offline service? → invoke aeval-offline-evaluation

When to Load Reference Files