ECS Observability
Advisory skill for designing the observability architecture of Amazon ECS workloads — which logging pipeline, which metrics stack, which tracing path, driven by the customer's compliance posture, existing tooling, scale, budget, and the launch types they actually run. This is primarily a design-and-selection brain: it produces recommendations and decision rationale, with AWS documentation URLs for every load-bearing fact — and it carries the diagnostic facts (the log-loss gate, ECS Exec constraints) that troubleshooting asks need, without being a step-by-step runbook.
The accuracy bar (non-negotiable for this skill). Launch-type scope-bleed is the #1 error class in this domain — capabilities documented for one launch type get silently generalized to all four. Every capability claim in this skill is scoped to EC2 / Fargate / Managed Instances (MI) / ECS Anywhere (EXTERNAL), and where AWS docs are silent the claim is marked "undocumented — verify". Never state launch-type support you cannot cite to an AWS-published source. When you can't ground a claim, say so — do not synthesize.
When to Use This Skill
Activate when the user wants to:
- Choose between CloudWatch-native (Container Insights, X-Ray, Application Signals), open-source-flavored (ADOT, Amazon Managed Prometheus, Amazon Managed Grafana), or third-party (Datadog, Splunk, SIEM via FireLens) observability for ECS
- Design log delivery — awslogs vs FireLens, blocking vs non-blocking mode, buffer sizing, multi-destination routing
- Decide standard vs enhanced Container Insights, or how to get GPU telemetry from ECS tasks
- Pick a tracing path for new or existing ECS services (ADOT/OTel vs legacy X-Ray SDK/daemon)
- Understand what observability each launch type (EC2, Fargate, Managed Instances, ECS Anywhere) can and cannot support
- Plan live-debugging access (ECS Exec) including private-subnet VPC endpoint requirements
- Weigh compliance requirements (no log loss, session auditing, KMS encryption) against availability and cost
Do NOT use this skill for:
- EKS or any Kubernetes observability → use
eks-best-practices(design guidance) oreks-operation-review(live-cluster audit), or the othereks-*skills - ECS deployment mechanics — circuit breaker configuration, rollback strategy, CI/CD pipelines → use
ecs-devops. Deployment-failure alerting and visibility (e.g., EventBridge onSERVICE_DEPLOYMENT_FAILED) stays here. - ECS security posture, IAM hardening, compliance audits beyond observability's audit-logging angle → use
ecs-security - Scoring or auditing a live ECS estate against a rubric → use
ecs-operation-review - Overall ECS architecture — compute selection, networking, service design → use
ecs-architect - GenAI/LLM workload design on ECS (GPU serving stacks, model hosting) → use
ecs-genai; come back here for the GPU telemetry question - Cost optimization of an existing observability bill as a FinOps exercise — this skill flags cost levers during design but does not audit spend
The Decision Framework
Elicit these five criteria before recommending a stack — each one prunes the option space:
| Criterion | Ask | Why it decides |
|---|---|---|
| Launch types in use | EC2, Fargate, Managed Instances, EXTERNAL — which, today and planned? | Hard eliminator. ADOT/AMP is unsupported on EXTERNAL; agentless GPU metrics are MI-only; Fargate forbids host agents; App Signals daemon strategy excludes Fargate. Check references/launch-type-matrix.md before every claim. |
| Compliance needs | Can any log record be lost? Are debug sessions audited? Customer-managed KMS? | Drives the blocking vs non-blocking log gate, FireLens filesystem buffering, ECS Exec session logging, and KMS endpoint/key-policy work. |
| Existing tooling | Grafana/PromQL estate? Datadog/Splunk/SIEM contract? OTel standardization? | Existing Grafana → ADOT+AMP+AMG. Third-party APM/SIEM → FireLens routing. No estate → CloudWatch-native is the lowest-friction default. |
| Scale | Log MB/s per task, task count, metric cardinality | High log throughput breaks default buffers (silent loss); container-level enhanced CI multiplies metric series; sidecar-per-task overhead compounds with task count. |
| Budget | Appetite for custom-metric, log-ingestion, and AMP/AMG spend | Free vended metrics + EventBridge events are the zero-cost floor; enhanced CI and high-cardinality Prometheus are the main cost levers. Link https://aws.amazon.com/cloudwatch/pricing/ — never quote prices. |
Default recommendation shape
For a CloudWatch-centric customer with no contrary criteria (EC2/Fargate fleets — EXTERNAL needs a different shape, since ADOT and Container Insights are not documented there): awslogs (with an explicit, documented delivery-mode decision) + Container Insights enhanced + ADOT sidecar for traces + EventBridge alerting on service action ERROR events. Enhanced is AWS's recommended Container Insights tier (per https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-ECS-cluster.html) — but it adds container-level metric cardinality, a cost lever to surface (Advisory Rule 3; cost posture in references/metrics-stacks.md). For MI fleets the logs + metrics + EventBridge parts carry over, but the ADOT-sidecar tracing leg is not documented for Managed Instances as of 2026-07-10 (the ADOT integration page enumerates Fargate + EC2 only, per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/application-metrics-prometheus.html) — verify against the live doc before recommending it there (see the matrix's ADOT row). Deviate per the criteria above.
Logs — the decision gate every engagement must surface
Facts verified 2026-07-09 against https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-account-settings.html and https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_LogConfiguration.html
Since June 25, 2025, the ECS default log delivery mode is non-blocking — and non-blocking mode silently drops logs when its in-memory buffer (default max-buffer-size: 10 MiB) fills. No metric, no daemon log line, no signal (per https://aws.amazon.com/blogs/containers/preventing-log-loss-with-non-blocking-mode-in-the-awslogs-container-log-driver/). The alternative, blocking, preserves every record but can hang the application and fail health checks if the log pipeline backs up. This is a compliance-versus-availability decision, and since the 2025 default change the risky posture is what customers get by not deciding:
- Availability-first (user-facing services, tolerable loss): keep
non-blocking, sizemax-buffer-sizeto ~25 MB for in-Region delivery at up to ~5 MB/s per container (AWS benchmark guidance — "no observed loss" in AWS's test runs, not a guarantee); above 6 MB/s AWS calls the driver's behavior "less predictable and consistent" — that becomes a FireLens conversation (see reference). Document the accepted loss risk. - Completeness-first (audit/financial/security logs): set
mode: blockingper container or set thedefaultLogDriverModeaccount setting — a per-Region setting, repeat it in every Region running tasks — and accept that a CloudWatch disruption can stall the app. - Both required: FireLens (Fluent Bit) with filesystem buffering — loss-resistant without blocking stdout/stderr.
- Always audit the effective account default first:
aws ecs list-account-settings --name defaultLogDriverMode --effective-settings.
FireLens is also the usual routing layer when the destination is not CloudWatch (Splunk, Datadog, OpenSearch, S3, Firehose, SIEMs) — though the native splunk log driver is documented on both Fargate and EC2 as a FireLens-free path to Splunk. FireLens is documented for Linux tasks on Fargate and EC2 only; MI and EXTERNAL support is undocumented — verify before advising.
For buffer benchmarks, FireLens rules, and high-throughput patterns, see: Log Delivery Reference
Metrics — Container Insights tiers and the GPU trap
Facts verified 2026-07-09 against https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html and https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html
- Standard Container Insights (
containerInsights=enabled): task/service/cluster-level metrics. Enhanced (enhanced, released Dec 2024): adds container-level metrics, per-TaskId dimensions, and health-status metrics — AWS's recommended default, at the cost of more metric series (cardinality is the main cost lever — link https://aws.amazon.com/cloudwatch/pricing/, never quote prices). Documented launch types: EC2 (agent ≥ 1.29), Fargate, Managed Instances; EXTERNAL is not listed — don't claim it. - GPU telemetry is the sharpest launch-type trap: agentless GPU/DCGM metrics (utilization, memory, power, temperature at container/task/instance level) are Managed Instances-ONLY and require enhanced CI (per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/monitoring-managed-instances.html). The EC2 launch type gets only the free GPU reservation metric — GPU utilization there means customer-managed DCGM/CloudWatch-agent tooling. Never generalize in either direction.
- Prometheus/Grafana route: ADOT sidecar remote-writes to Amazon Managed Service for Prometheus, visualized in Amazon Managed Grafana — documented for Fargate and EC2 only; EXTERNAL explicitly unsupported, MI not documented as of 2026-07-10 (see the ADOT row of references/launch-type-matrix.md for the source quote) — verify before advising MI. Alternative keeping data in CloudWatch: the CloudWatch agent's Prometheus scrape support.
- Zero-cost floor available everywhere: free vended CPU/memory metrics + EventBridge service/task/deployment events + container health checks.
For metric tables, instance-level collection, and AMP vs CloudWatch-agent-Prometheus selection, see: Metrics Stacks Reference
Traces — OTel is the path; X-Ray SDK/daemon is legacy
Facts verified 2026-07-10 against https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-daemon-timeline.html
- The X-Ray SDKs and daemon entered maintenance mode on February 25, 2026 (security fixes only); AWS recommends OpenTelemetry. Steer all new tracing to the ADOT sidecar collector (
public.ecr.aws/aws-observability/aws-otel-collector), which still delivers traces to the X-Ray backend — the maintenance-mode timeline names only the SDKs and daemon, not the backend service (inference from https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-daemon-timeline.html, which covers SDKs/daemon only; the migration guide keeps X-Ray as the trace destination). - CloudWatch Application Signals on ECS is custom setup with no autodiscovery — service names are wired via env vars. Sidecar strategy: EC2 + Fargate, requires
awsvpc. Daemon strategy: one agent per cluster, excludes Fargate. MI: not documented for either strategy as of 2026-07-10; EXTERNAL: sidecar structurally excluded (awsvpcrequired), daemon undocumented (see the App Signals rows of the matrix). - Placement rule: Fargate → sidecar only (no host access); EC2 → sidecar or daemon-scheduled collector; MI → neither pattern is documented as of 2026-07-10 (the ADOT page enumerates Fargate + EC2 only — per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/application-metrics-prometheus.html; verify against the live doc before advising, see the matrix's ADOT row); EXTERNAL → ADOT unsupported.
For migration gotchas, IAM, and Application Signals strategy details, see: Tracing and Signals Reference
Live debugging — ECS Exec
Facts verified 2026-07-09 against https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html
- Works on all four launch types (one of the few capabilities that does), and is the sole shell path on Managed Instances (no SSH there).
- Private-subnet precision, for tasks in a VPC (EC2/Fargate/MI): ECS Exec needs the
ssmmessagesinterface VPC endpoint — not the fullssm/ec2messagestrio (per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html) — plus a KMS endpoint only when sessions are encrypted with a customer-managed key. On EXTERNAL (ECS Anywhere) the host must reachssm+ec2messages+ssmmessagesregardless — an SSM Agent registration prerequisite (per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere.html). - Compliance anchors: session logging via cluster
executeCommandConfiguration, CloudTrailExecuteCommandevents, and an IAM deny onssm:StartSessionto close the unlogged side door. Commands run as root;readonlyRootFilesystemunsupported.
For full constraints and audit setup, see: Native Visibility and ECS Exec Reference
Launch-Type Discipline
Before asserting any capability, check references/launch-type-matrix.md — the per-capability EC2/Fargate/MI/EXTERNAL matrix with a source URL per row. The matrix is the single source of truth for these facts; the recurring traps (restated here per the matrix, verified 2026-07-10 — re-check the matrix row before load-bearing use):
- Agentless GPU/DCGM → MI only (EC2 gets reservation-only).
- ADOT/AMP → documented for Fargate + EC2; EXTERNAL explicitly unsupported; MI undocumented — verify before advising.
- FireLens → Linux Fargate/EC2 documented; MI/EXTERNAL undocumented — verify before advising.
- Application Signals daemon → no Fargate; sidecar → requires
awsvpc(which EXTERNAL cannot run). - EXTERNAL supports bridge/host/none networking only — anything demanding
awsvpcis structurally out. - Fargate has no host — every agent is an in-task sidecar; MI's AMI is AWS-owned — nothing can be baked into it.
How to Use the References
This skill uses progressive disclosure — essential decision guidance is in this file; detailed reference material is loaded on demand:
| Reference | Load when the task is about… |
|---|---|
| log-delivery.md | blocking vs non-blocking details, buffer-size benchmarks, awslogs options, FireLens/Fluent Bit configuration, filesystem buffering, high-throughput logging, routing to third-party destinations (Datadog, Splunk, SIEMs) and awslogs-vs-FireLens selection |
| metrics-stacks.md | Container Insights standard vs enhanced metric tables, GPU/DCGM scoping, instance-level metrics, ADOT+AMP+AMG vs CloudWatch-agent Prometheus, metric cost levers |
| tracing-and-signals.md | X-Ray maintenance-mode details and migration, ADOT collector setup and IAM, Application Signals sidecar vs daemon, language support |
| launch-type-matrix.md | any "does X work on launch type Y" question — check before every capability claim |
| native-visibility-and-exec.md | EventBridge events, service events, container health checks, ECS Exec constraints and VPC endpoints, monitoring-plan baseline |
Advisory Rules
- Scope every capability claim by launch type. If the matrix row says "undocumented — verify", say exactly that to the customer and check the live AWS doc.
- Always surface the log delivery-mode gate in any logging discussion — the customer must consciously choose a failure mode; the post-2025 default chooses silent loss for them.
- Never quote prices or assert the enhanced-CI pricing model for ECS — link https://aws.amazon.com/cloudwatch/pricing/ instead.
- Do not recommend X-Ray SDK/daemon for new builds — ADOT/OTel, always, since the 2026-02-25 maintenance date.
- Recommend the zero-cost floor first (vended metrics, EventBridge ERROR-event alerting, health checks) — it survives every stack decision and every launch type.
- Cite as you advise — every load-bearing recommendation carries its AWS documentation URL, dated where the fact is volatile.
Sources
- ECS account settings — default log driver mode · LogConfiguration API reference · Using the awslogs driver · Preventing log loss with non-blocking mode (AWS Containers blog)
- Using FireLens · High-throughput log configuration
- Container Insights for ECS · Setting up Container Insights on ECS · Enhanced observability metrics for ECS · Container Insights enhanced observability What's New (Dec 2024)
- Monitoring Managed Instances (incl. agentless DCGM) · ECS Managed Instances · Free vended CloudWatch metrics
- ADOT/AMP for ECS application metrics · ECS trace data via ADOT · ADOT ECS setup · AMP query onboarding · CloudWatch agent Prometheus for ECS
- X-Ray daemon on ECS · X-Ray SDK/daemon maintenance timeline · X-Ray to OTel migration
- Application Signals on ECS · Sidecar strategy · Daemon strategy · Support matrix
- ECS Exec · Session Manager PrivateLink
- ECS EventBridge events · Service action events · Service event messages · Container health checks · ECS monitoring overview · Recommended alarms
- ECS Anywhere · CloudWatch pricing · CloudWatch cost guidance
- Structural framing borrowed with citation (not forked) from aws/agent-toolkit-for-aws
skills/core-skills/aws-containersandaws-observability@ commit 43e9d50.
This skill is provided as sample code for educational and demonstration purposes only. Recommendations should be validated against current AWS documentation before acting on them. See the project's README and LICENSE for full terms.
