Datadog APM
Distributed tracing, service maps, and performance analysis.
Requirements
Datadog Labs Pup should be installed via:
bashbrew tap datadog-labs/pack brew install pup
Quick Start
bashpup auth login pup apm services list --env production pup traces search --query="service:api-gateway" --from="1h"
Services
List Services
--env is required for all apm services commands.
bashpup apm services list --env production pup apm services list --env staging
Service Statistics
bashpup apm services stats --env production pup apm services stats --env production --from 4h
Service Operations and Resources
bash# List operations for a service pup apm services operations --env production --service api-gateway # List resources (endpoints) for an operation pup apm services resources --env production --service api-gateway --name http.request
Service Dependencies
bashpup apm dependencies list --env production
Flow Map
bash# View service flow map (--query and --env required) pup apm flow-map --query "service:api-gateway" --env production
Traces
Traces are searched via the top-level traces command (not under apm).
Important: APM durations are in nanoseconds: 1 second = 1,000,000,000 ns.
Search Traces
bash# By service pup traces search --query="service:api-gateway" --from="1h" # Errors only pup traces search --query="service:api-gateway status:error" --from="1h" # Slow traces (>1 second = 1000000000 ns) pup traces search --query="service:api-gateway @duration:>1000000000" --from="1h" # With specific tag pup traces search --query="service:api @http.url:/api/users" --from="1h"
Aggregate Traces
bash# Average duration by resource pup traces aggregate \ --query="service:api-gateway" \ --compute="avg(@duration)" \ --group-by="resource_name" \ --from="1h" # Error count by service pup traces aggregate \ --query="status:error" \ --compute="count" \ --group-by="service" \ --from="1h" # p99 latency pup traces aggregate \ --query="service:api-gateway" \ --compute="percentile(@duration, 99)" \ --from="1h"
Key Metrics
| Metric | What It Measures |
|---|---|
trace.http.request.hits | Request count |
trace.http.request.duration | Latency |
trace.http.request.errors | Error count |
trace.http.request.apdex | User satisfaction |
⚠️ Trace Sampling
Not all traces are kept. Understand sampling:
| Mode | What's Kept |
|---|---|
| Head-based | Random % at start |
| Error/Slow | All errors, slow traces |
| Retention | What's indexed (billed) |
Trace Retention Costs
| Retention | Cost |
|---|---|
| Indexed spans | $$$ per million |
| Ingested spans | $ per million |
Best practice: Only index what you need for search.
Service Level Objectives
Link APM to SLOs:
bashpup slos create --file slo.json
Common Queries
| Goal | Query |
|---|---|
| Slowest endpoints | pup traces aggregate --query="service:api" --compute="avg(@duration)" --group-by="resource_name" --from="1h" |
| Error rate by service | pup traces aggregate --query="status:error" --compute="count" --group-by="service" --from="1h" |
| Throughput | pup traces aggregate --query="service:api" --compute="count" --group-by="resource_name" --from="1h" |
Service Config
Query service instance metadata — instance IDs, hostnames, and config IDs for all running instances of a service. Returns up to 100 instances.
bash# Get instance metadata for a service pup apm service-config get --service-name my-service # Filter by environment pup apm service-config get --service-name my-service --env prod # Filter by specific instance IDs pup apm service-config get --service-name my-service --service-instance-ids "id-1,id-2"
Note on service identity:
service_nameandenvcome from the SDK telemetry pipeline and may differ from values in the Service Catalog.
Service Library Config
Query the APM tracer configuration deployed across all running instances of a service. Useful for auditing config drift — finding instances where tracing, profiling, or AppSec is misconfigured relative to the rest of the fleet.
bash# Get tracer config for a service pup apm service-library-config get --service-name my-service # Filter by environment pup apm service-library-config get --service-name my-service --env prod # Filter by language pup apm service-library-config get --service-name my-service --env prod --language python # Only show configs where instances disagree (config drift) pup apm service-library-config get --service-name my-service --mixed
Note on service identity:
service_name,env, andlanguage_namecome from the SDK telemetry pipeline and reflect what the tracer reports at runtime. These may differ from values in the Service Catalog, which aggregates data from multiple sources (APM spans, USM, infrastructure tags, manual definitions).
Troubleshooting
| Problem | Fix |
|---|---|
| No traces | Check ddtrace installed, DD_TRACE_ENABLED=true |
| Missing service | Verify DD_SERVICE env var |
| Traces not linked | Check trace headers propagated |
| High cardinality | Don't tag with user_id/request_id |
--env required error | Always pass --env to apm services commands |
