Dynatrace

Configure, troubleshoot, or investigate incidents in Dynatrace.

Mode: setup

Deploy the Dynatrace Operator and OneAgent on Kubernetes.

Steps:

1. Ask for: environment ID, Kubernetes distribution (EKS/AKS/GKE), monitoring mode (fullStack / cloudNativeFullStack)
2. Install Operator from a pinned release, never releases/latest: kubectl apply -f .../dynatrace-operator/releases/download/v1.11.0/kubernetes.yaml
3. Create Secret with apiToken and dataIngestToken (store in Kubernetes Secret or secrets manager — never plain values)
4. Generate DynaKube CR with cloudNativeFullStack for automatic injection — no pod restarts required
5. Enable metadataEnrichment: true for Kubernetes metadata on all telemetry
6. Verify injection: kubectl describe pod <app-pod> | grep dynatrace

Required token scopes:

• apiToken: ReadConfig WriteConfig DataExport LogExport ReadSyntheticData WriteAnomalyDetection
• dataIngestToken: metrics.ingest logs.ingest

Reference: references/dynatrace.md → Deployment, Token Scopes

Mode: instrument

Add custom spans or business transaction tracing to a service.

Steps:

1. OneAgent auto-instruments HTTP, database, cache, and messaging — code changes only needed for custom business spans
2. Ask for: language (Node.js / Python / Java), business operations to trace (payment, checkout, data pipeline)
3. Generate SDK code for custom spans using @dynatrace/oneagent-sdk (Node.js), oneagent-sdk (Python), or OneAgent Java SDK
4. For cross-service propagation: forward and extract the x-dynatrace header between services
5. Verify in Distributed Traces UI: service appears in Service Map with custom spans

Reference: references/dynatrace.md → Code-Level Instrumentation

Mode: monitor

Configure anomaly detection and alerting for a service.

Steps:

1. Ask for: service entity ID (from Smartscape or Settings API), alerting thresholds (auto-detection or fixed), notification target
2. Generate Terraform dynatrace_service_anomalies_v2 resource for failure rate and response time
3. Generate Terraform dynatrace_alerting profile linking anomalies to the team
4. Davis AI will auto-detect baseline anomalies — custom thresholds override only when auto-detection is too noisy
5. Wire alerting profile to notification integration (PagerDuty, Slack, Opsgenie)

Reference: references/dynatrace.md → Terraform Provider, Davis AI Problem Feeds

Mode: slo

Define a Dynatrace SLO.

Steps:

1. Ask for: SLI expression (availability / latency / custom), target %, timeframe
2. Generate Terraform dynatrace_slo_v2 resource
3. Use built-in metrics for availability: builtin:service.errors.server.successCount / builtin:service.requestCount.server
4. Validate the metric expression returns data in the Metrics Explorer before applying
5. Set both target_success and target_warning thresholds

Reference: references/dynatrace.md → SLOs

Mode: dashboard

Create a Dynatrace dashboard.

Steps:

1. Ask for: service entity ID, key metrics to show (availability, latency, throughput, error rate)
2. Generate Terraform dynatrace_json_dashboard resource pointing to a JSON dashboard file
3. Use built-in service metrics: builtin:service.requestCount.server, builtin:service.response.time, builtin:service.errors.server.rate
4. Provide dashboard JSON with tiles for each metric using DATA_EXPLORER tile type

Reference: references/dynatrace.md → Terraform Provider

Mode: investigate

Live incident investigation using the Dynatrace MCP server.

Requires the Dynatrace MCP server connected to Claude Code. See setup in references/dynatrace.md → MCP Server Setup.

Cost note: execute_dql queries scan Grail data and may incur costs based on your Dynatrace consumption model. Start with short timeframes (last 1h–24h). Set DT_GRAIL_QUERY_BUDGET_GB to cap session spend.

Phase 1 — Triage (what is Davis AI seeing?)

Ask Claude to run these via the MCP server:

• List all open Problems — Davis AI auto-detects anomalies and groups related symptoms
• Fetch full problem details including root cause entity, impact, and affected services
• Check recent Kubernetes events for the affected namespace

Copy code
List all open Problems in Dynatrace right now.
Get full details for Problem <problem-id> including root cause and affected entities.
Show me Kubernetes events for namespace production in the last 30 minutes.

Phase 2 — Signals (logs, traces, exceptions)

Use DQL via the MCP to query Grail directly:

• Fetch error logs for the affected service in the incident window
• List recent exceptions with stack traces
• Find distributed traces with errors to identify the failing call

Copy code
Show me error logs for the orders-service in the last hour.
List the top exceptions for orders-service with stack traces.
Find distributed traces with errors for orders-service — show the slowest and most frequent.

Example DQL the MCP can generate and execute:

dqlCopy code
fetch logs
| filter service.name == "orders-service" and loglevel == "ERROR"
| sort timestamp desc
| limit 50
| fields timestamp, content, trace_id, span_id

dqlCopy code
fetch spans
| filter service.name == "orders-service" and status == "ERROR"
| sort timestamp desc
| limit 20
| fields timestamp, span_name, error.message, trace_id, duration

Phase 3 — Root cause with Davis AI

Let Davis AI perform automated analysis:

• Ask Davis Copilot to explain the problem in plain English
• Run a Davis Analyzer on the affected service for automated root cause analysis
• Find the entity (service, host, process group) at the root of the problem

Copy code
Chat with Davis Copilot: "Why is orders-service having elevated error rates since 14:00 UTC?"
List available Davis analyzers for root cause analysis.
Find the entity named "orders-service" and show its current health.

Phase 4 — Resolution and follow-up

• Create a Dynatrace Notebook capturing the timeline, DQL queries, and evidence
• Send a Slack message or email via the MCP notification tools
• Close the Problem with a resolution note once fix is validated

Copy code
Create a Dynatrace notebook summarising the orders-service incident with the DQL queries we ran.
Send a Slack message to #incidents: "orders-service error rate normalising, fix deployed at <time>."
Close Problem <problem-id> with message "Root cause: bad deploy at 14:05 UTC, rolled back at 14:22 UTC."

Reference: references/dynatrace.md → MCP Server Setup, Incident Investigation Workflow

Mode: debug

Diagnose Dynatrace data gaps or injection failures without the MCP server.

Classify the failure:

• Injection — OneAgent not injecting into pods
• Traces — service not in Service Map or distributed traces broken
• Metrics — custom metrics not appearing
• SLO — SLO shows 0% or "No data"
• Davis AI — Problems not being created for known issues

Evidence to collect:

bashCopy code
# Operator and DynaKube status
kubectl -n dynatrace get dynakube
kubectl -n dynatrace get pods

# Check injection on app pod
kubectl describe pod <app-pod> | grep -i dynatrace

# Verify MINT ingestion (custom metrics)
curl "https://{env}.live.dynatrace.com/api/v2/metrics/query?metricSelector=<your.metric>" \
  -H "Authorization: Api-Token ${DT_API_TOKEN}"

# List open Davis AI problems
curl "https://{env}.live.dynatrace.com/api/v2/problems?problemSelector=status(OPEN)" \
  -H "Authorization: Api-Token ${DT_API_TOKEN}"

Provide: symptom → root cause hypothesis → evidence command → fix → validation step