Observability

Set up or improve observability for a service or platform component.

Interactive Wizard (fires when no arguments are provided)

When invoked with no arguments, ask before proceeding:

Q1 — Mode?

Copy code
What do you need?
  1. instrument — add structured logs, Prometheus metrics, and OTel tracing to a service
  2. dashboard  — create a Grafana RED/USE dashboard for a service
  3. alert      — write Prometheus alerting rules for a service
  4. slo        — define SLIs, error budgets, and SLO burn-rate alerts
  5. loadtest   — write and run a k6 load test
  6. capacity   — estimate resource requirements and HPA configuration

Enter 1–6 or mode name:

Q2 — Context (after mode selected):

• instrument: What language/framework and which metrics/tracing backend (Prometheus, Datadog, Jaeger, Tempo)?
• dashboard: Service name and which signal to lead with — request-based (RED) or resource-based (USE)?
• alert: Service name and what SLIs matter most — error rate, latency, availability?
• slo: Service name, expected availability target (e.g. 99.9%), and current p95 latency baseline:
• loadtest: Target endpoint, expected peak RPS, and SLO thresholds (p95 latency, max error rate):
• capacity: Expected peak RPS, measured p99 latency at current load, and memory per pod:

Mode: instrument

Add the three pillars — logs, metrics, traces — to a service.

Steps:

1. Ask for: language/framework, existing logging library (if any), metrics backend (Prometheus / Datadog / CloudWatch), tracing backend (Jaeger / Tempo / OTLP)
2. Add structured JSON logging with correlation IDs (Pino for Node.js, structlog for Python)
3. Instrument RED metrics: http_requests_total counter, http_request_duration_seconds histogram, per route and status
4. Add OpenTelemetry tracing with span attributes on critical paths
5. Expose /metrics scrape endpoint (Prometheus) or configure push exporter
6. Add /healthz and /readyz health check endpoints
7. Document what NOT to log (passwords, tokens, PII)

Reference: references/observability.md → Structured Logging, Prometheus Metrics, OpenTelemetry Tracing

→ Next: Run /platform-skills:observability alert to write alerting rules for the metrics just added, then /platform-skills:observability slo to wrap them in an error budget.

Mode: dashboard

Create a Grafana dashboard for a service.

Steps:

1. Choose method: RED (request-based services) or USE (resource-based infrastructure)
2. Define panels: request rate, error rate %, p50/p95/p99 latency, active connections, queue depth
3. Set meaningful Y-axis units (req/s, ms, %)
4. Add threshold lines at SLO boundaries
5. Configure template variables for environment and service filtering

Reference: references/observability.md → Grafana Dashboards

→ Next: Run /platform-skills:observability slo to add SLO burn-rate alerts and an error budget panel to this dashboard.

Mode: alert

Write Prometheus alerting rules for a service.

Steps:

1. Identify SLIs: error rate, latency percentiles, availability
2. Write alert expressions using rate() over 5m windows
3. Set for: duration ≥ 1m to suppress transient noise
4. Add severity label (critical / warning) and runbook annotation to every alert
5. Validate: no alert fires on healthy baseline, alert fires on injected fault

Alert design rules:

• Page on symptoms (error rate, latency), not causes (CPU %)
• Every alert needs a runbook URL
• Derive SLO burn-rate alerts from error budget, not raw thresholds

Reference: references/observability.md → Alerting Rules

Security posture and CVE alerts from Kubescape's exporter (kubescape_controls_total_*, kubescape_vulnerabilities_*; failing scans show only in the scanner's logs, not in these metrics) → /platform-skills:kubescape-operator metrics.

→ Next: Run /platform-skills:observability slo to promote these symptom alerts to proper SLO burn-rate alerts backed by an error budget.

Mode: loadtest

Write and run a k6 load test.

Steps:

1. Ask for: target endpoint, expected peak RPS, SLO thresholds (p95 latency, error rate)
2. Write ramp-up → steady-state → ramp-down stages
3. Set thresholds matching the SLO
4. Add check() assertions on status code and response time

Version check: k6 --version — must be ≥ 1.0.0. k6 v0.x and v1.x have incompatible JavaScript API syntax. Upgrade: brew install k6 (macOS) or see https://grafana.com/docs/k6/latest/set-up/install-k6/ for other platforms.

5. Run: k6 run --out json=results.json load-test.js
6. Interpret results: p95/p99 latency, error rate, throughput achieved vs. thresholds

Reference: references/observability.md → Load Testing

Mode: capacity

Estimate resource requirements and HPA configuration for a service.

Steps:

1. Gather: expected peak RPS, measured p99 latency, memory per pod at current load
2. Apply formula: replicas = ceil((peak_rps × avg_latency_s) / target_concurrency_per_pod)
3. Add 50% headroom for spikes
4. Generate HPA manifest with CPU utilisation target ≤ 60%
5. Set resource requests to measured baseline + 20%; set memory limit; omit CPU limit unless throttling is acceptable
6. Define min/max replica bounds

Reference: references/observability.md → Capacity Planning

→ Next: After sizing, run /platform-skills:observability loadtest to validate the HPA triggers correctly under synthetic load.

Mode: slo

Define SLIs, set error budgets, and generate SLO burn-rate alerts from first principles.

Steps:

1. Define SLIs — identify what "good" looks like for this service:

   • Availability: sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
   • Latency: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) < 0.3
   • Choose one primary SLI per service; add a secondary only if the first doesn't capture the user journey

2. Set the SLO target — start conservative, tighten over time:

   Copy code
   SLO:           99.9% availability over a 30-day rolling window
   Error budget:  0.1% = 43.2 minutes per 30 days
   

3. Generate burn-rate alerts — multiwindow, multi-burn-rate (Google SRE Book approach):

   yamlCopy code
   # Fast burn — consumes 5% of monthly budget in 1h → page immediately
   - alert: SLOFastBurn
     expr: |
       (
         rate(http_requests_total{status=~"5.."}[1h])
         / rate(http_requests_total[1h])
       ) > (14.4 * 0.001)   # 14.4× burn rate exhausts budget in ~2 days
     for: 2m
     labels:
       severity: critical
     annotations:
       summary: "Fast error budget burn on {{ $labels.service }}"
       runbook: "https://runbooks.internal/slo-fast-burn"
   
   # Slow burn — consumes 10% of monthly budget in 6h → ticket
   - alert: SLOSlowBurn
     expr: |
       (
         rate(http_requests_total{status=~"5.."}[6h])
         / rate(http_requests_total[6h])
       ) > (6 * 0.001)      # 6× burn rate exhausts budget in ~5 days
     for: 15m
     labels:
       severity: warning
     annotations:
       summary: "Slow error budget burn on {{ $labels.service }}"
       runbook: "https://runbooks.internal/slo-slow-burn"
   

4. Track remaining error budget — add to the Grafana dashboard:

   promqlCopy code
   # Error budget remaining (%) over 30d
   1 - (
     sum(increase(http_requests_total{status=~"5.."}[30d]))
     / sum(increase(http_requests_total[30d]))
   ) / 0.001   # divide by error budget fraction (1 - SLO target)
   

5. Error budget policy — state in writing what happens when the budget is depleted:

   • 50% remaining → normal feature velocity

   • 10–50% remaining → reliability improvements prioritised alongside features
   • <10% remaining → feature freeze; only reliability work until budget recovers

Key rules:

• Alert on burn rate, not on raw error count — burn rate is predictive
• The for: on fast burn should be short (1–2m); slow burn needs longer (10–15m)
• Latency SLOs need histogram buckets aligned to the SLO threshold at chart creation time — you cannot retroactively add buckets

→ Next: Run /platform-skills:observability alert to add symptom-based alerts alongside SLO burn-rate alerts.

Common mistakes

• Alerting on CPU% or memory% — these are causes, not symptoms. Alert on error rate and latency; let CPU/memory be dashboard panels only
• Missing for: duration — without it, a single bad scrape triggers a page. Minimum 1m for slow alerts, 2m for fast SLO burn
• Not defining SLOs before writing alerts — without an SLO, you don't know what threshold to alert at. Define the SLO first (slo mode), then write alerts
• Histogram bucket boundaries misaligned with SLO — if your SLO is p95 < 300ms and you have no bucket at 0.3s, histogram_quantile interpolates inaccurately. Set le values at 0.1, 0.25, 0.3, 0.5, 1.0, 2.5
• Labelling with high-cardinality dimensions — never use user_id, session_id, or request_id as Prometheus label values. Cardinality explodes memory usage
• Alert fatigue from symptom + cause alerts — if you alert on both "error rate high" and "database connection pool exhausted", both fire simultaneously for the same incident. Alert on the symptom; include cause investigation in the runbook