Huawei Cloud Agentarts Obs Diagnosis

Huawei Cloud AgentArts Observability No-Data Diagnosis

Overview

This skill systematically diagnoses missing metrics, logs, and traces/sessions in Huawei Cloud AgentArts agent-ops platform. It follows a three-phase diagnostic flow: subscription metadata check → data pipeline troubleshooting → query parameter validation, covering three data sources: APM (traces/metrics), AOM (metrics), and LTS (logs).

Architecture

Copy code
Application (OTel SDK/Collector)
  │
  ├── Traces:  OTEL SDK → APM (OTLP) → deliverConfig → Kafka → agent-ops
  ├── Metrics: Application → AOM (Prometheus) → agent-ops
  └── Logs:    Application → LTS → agent-ops (requires logGroup/logStream metadata)

Data Sources and Metadata

Prerequisites

1. Huawei Cloud Credentials

bashCopy code
export HUAWEI_ACCESS_KEY="<your-access-key-id>"
read -rs HUAWEI_SECRET_KEY; export HUAWEI_SECRET_KEY
export HUAWEI_REGION="cn-north-4"

2. hcloud CLI (KooCLI)

Install: https://support.huaweicloud.com/qs-hcli/hcli_02_003.html

3. IAM Permissions

See references/iam-policies.md for least-privilege policy.

Workflow

Step 1: Identify Missing Data Type

Use AskUserQuestion to determine which data type is missing:

Record symptoms: complete vs partial absence, error codes, onset time.

Step 2: Confirm Environment Context

Step 3: Check Subscription Metadata (Phase 1 — Always First)

Missing subscription metadata is the most common root cause of no-data issues.

Check APM business and token:

bashCopy code
hcloud APM ListBusiness --cli-region=cn-north-4

If no business exists → APM not subscribed. Create business:

bashCopy code
hcloud APM CreateBusiness --cli-region=cn-north-4 \
  --name="obs-diagnosis-check" \
  --display_name="obs-diagnosis-check" \
  --descp="diagnosis" \
  --cmdb_datasource_type=OTEL

Verify business token:

bashCopy code
hcloud APM ShowToken --cli-region=cn-north-4 \
  --x-business-id={business_id}

Check AOM Prometheus instance and access code:

bashCopy code
hcloud AOM ListPromInstance --cli-region=cn-north-4 --Enterprise-Project-Id=0
hcloud AOM ListAccessCode --cli-region=cn-north-4

If no PromInstance → AOM not configured. Create Prometheus instance (--prom_type must be one of ECS/CCE/REMOTE_WRITE/CLOUD_SERVICE/ACROSS_ACCOUNT; --region header is required and must match --cli-region):

bashCopy code
hcloud AOM CreatePromInstance --cli-region=cn-north-4 \
  --prom_name="obs-diagnosis" \
  --prom_type=REMOTE_WRITE \
  --region=cn-north-4

Check LTS log groups and streams:

bashCopy code
hcloud LTS ListLogGroups --cli-region=cn-north-4
hcloud LTS ListLogStreams --cli-region=cn-north-4 --log_group_id={group_id}

If no logGroup/logStream → LTS not subscribed for agent-ops.

Step 4: Diagnose by Data Type (Phase 2 — Pipeline Check)

4A. Metrics No-Data Diagnosis

Check pipeline bottom-up:

1. Application side: Is OTEL metrics SDK / Prometheus exporter configured?
2. AOM side:

   bashCopy code
   hcloud AOM ListPromInstance --cli-region=cn-north-4 --Enterprise-Project-Id=0
   hcloud AOM ListAccessCode --cli-region=cn-north-4
   hcloud AOM ListMetadataAomPromGet --cli-region=cn-north-4
   

   • Prometheus instance valid?
   • Access code valid?
   • Endpoint reachable?
3. Query side: Use Prometheus query to verify data exists:

   bashCopy code
   hcloud AOM ListInstantQueryAomPromGet --cli-region=cn-north-4 \
     --query="up" \
     --prom_instance_id={prom_id}
   

4. agent-ops filter: Check metric_name matches (resource_tokens, model_tokens, session_tokens, trace_tokens), filter labels match (gen_ai_resource_type, resource_id, model_id).

4B. Logs No-Data Diagnosis

1. Application side: Is ICAgent / OTEL logs exporter configured?
2. LTS side:

   bashCopy code
   hcloud LTS ListLogGroups --cli-region=cn-north-4
   hcloud LTS ListLogStreams --cli-region=cn-north-4 --log_group_id={group_id}
   hcloud LTS ListLogs --cli-region=cn-north-4 \
     --log_group_id={group_id} \
     --log_stream_id={stream_id} \
     --start_time=1720000000000 \
     --end_time=1720600000000
   

3. Query parameters: Check agentRunId, agentId, traceId, logType, keyword syntax (LTS query language).
4. Error codes:
   • LTS.2446 → Unsupported query syntax — fix keyword syntax
   • 403 → Permission denied — check agency token (ops_admin_trust)

4C. Traces/Sessions No-Data Diagnosis

1. Application side:

   bashCopy code
   hcloud APM SearchAgent --cli-region=cn-north-4 --x-business-id={business_id}
   

   Is OTEL SDK configured with correct endpoint and token?
2. APM deliverConfig:
   • Is deliver_otel_trace=true in deliverConfig?
   • Is Kafka reachable (bootstrap-servers, SASL_SSL, topic exists)?
3. APM data verification:

   bashCopy code
   hcloud APM SearchTransaction --cli-region=cn-north-4 \
     --x-business-id={business_id} \
     --start_time=1720000000000 \
     --end_time=1720600000000
   

4. agent-ops query side (API only — not in KooCLI):

   bashCopy code
   curl -X POST ${AGENT_OPS_ENDPOINT}/v1/ops/observation/traces \
     -H "Authorization: Bearer ${IAM_TOKEN}" \
     -H "Content-Type: application/json" \
     -d '{"start_time": 1720000000000, "end_time": 1720600000000}'
   

Step 5: Common Checks (All Data Types)

1. Timestamp format: Must be 13-digit milliseconds (10-digit seconds → no data)
2. Region consistency: Ingestion region must match query region
3. Authentication: IAM token / AppCode valid? 401 → expired, 403 → insufficient permission. ${IAM_TOKEN} 需在运行期由 IAM token / AppCode 获取后替换（见上 Step 4 agent-ops 查询命令）；未替换则为占位符，照抄请求必返回 401。
4. Permission check:

   bashCopy code
   hcloud IAM ListPoliciesV5 --cli-region=cn-north-4
   

Step 6: Generate Diagnostic Report

See references/diagnostic-report-template.md for the report format.

Core Commands

Parameter Confirmation

KooCLI Command Format Standard

bashCopy code
hcloud <Service> <Operation> --cli-region=<region> [--key=value ...]

Reference Documents

• references/iam-policies.md — Least-privilege IAM policies
• references/cli-installation-guide.md — hcloud CLI installation guide
• references/verification-method.md — Verification methods and API status
• references/dataflow-diagram.md — Mermaid data flow and diagnostic flow
• references/acceptance-criteria.md — Acceptance criteria for diagnosis
• references/diagnostic-report-template.md — Diagnostic report template

Edge Cases

Notes

• Priority order: Subscription metadata → data pipeline → query parameters → permissions
• Most common root cause: Missing subscription metadata (apmBusiness, promInstance, logGroup)
• Timestamp pitfall: 10-digit seconds instead of 13-digit milliseconds
• Credential security: Never hardcode AK/SK. Read from env vars or CLI profile.
• Least privilege: See references/iam-policies.md for required permissions.