Msk Operations

Amazon MSK Operations

Operate, troubleshoot, and assess Amazon MSK (Managed Streaming for Apache Kafka) Provisioned clusters — both Standard and Express broker types. This skill covers day-to-day operations (health assessments, monitoring setup) and ad-hoc incident response (performance degradation, consumer lag, storage full, unexpected broker reboots).

When to Use

Activate this skill when the user asks to:

• Review, audit, or assess an MSK cluster for best practices, health, or operational readiness.
• Troubleshoot an MSK cluster problem: high CPU, high produce/fetch latency, consumer lag, broker storage running out, TrafficShaping events, under-replicated partitions, or an unexpected broker restart.
• Set up MSK monitoring: choose a monitoring level, create recommended CloudWatch alarms and dashboards, understand the metrics available in the AWS/Kafka namespace.
• Plan an MSK maintenance event: rolling restart, Kafka version upgrade, security patching, broker instance type change.
• Advise on Kafka client (producer / consumer) configuration when the client is connecting to an MSK cluster.

Do not activate this skill for MSK Connect, MSK Serverless, or MSK Replicator — those are separate services with their own operational surfaces.

Broker Type Determination

Determine the broker type first — many checks differ between Standard and Express.

Copy code
aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>

Check ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType:

• Starts with kafka. (e.g. kafka.m5.large, kafka.m7g.xlarge) → Standard broker.
• Starts with express. (e.g. express.m7g.large) → Express broker.

Key Standard vs Express differences

Standard brokers use customer-managed EBS volumes for storage. You choose instance types (kafka.m5.*, kafka.m7g.*), provision EBS, and manage storage scaling. Standard brokers have scheduled maintenance windows.

Express brokers provide fully managed, pay-as-you-go storage with no EBS provisioning. Instance types are prefixed with express.m7g.*. Express brokers offer up to 3× more throughput per broker than Standard, and have no maintenance windows. Express enforces a fixed replication factor of 3 and min.insync.replicas=2 — you cannot create topics with RF=1.

Critical Warnings

• This skill is read-only. Every command in this file and in references/ that mutates cluster state — update-broker-storage, create-configuration, update-cluster-configuration, update-monitoring, put-metric-alarm, reboot-broker, and any partition reassignment — is a recommendation for the operator to run after review. Present these as proposed remediations with expected impact and preconditions; do NOT execute them, and do NOT imply that the agent will run them.
• NEVER reboot brokers while UnderReplicatedPartitions > 0 (Standard only — Express brokers do not emit URP). This risks data loss and extended outages.
• NEVER recommend partition reassignment without first checking replication status. Reassignment during URP compounds the problem.
• linger.ms=0 is the #1 cause of "high CPU" on MSK. ALWAYS check client batch configuration before recommending broker scaling.
• EBS throughput ceilings are invisible in Kafka metrics — ALWAYS check EBS volume metrics (VolumeReadBytes, VolumeWriteBytes, VolumeQueueLength) when diagnosing Standard broker latency.
• Express brokers have NO customer-managed EBS — do NOT recommend EBS expansion or provisioned EBS throughput for Express clusters.
• Express brokers enforce fixed RF=3 and min.insync.replicas=2 — do NOT attempt to create topics with RF=1 on Express. If RF=1 is needed, use Standard brokers.

Quick Diagnostics

These five checks cover the most common MSK issues. Use them before loading a reference file.

1. CpuUser + CpuSystem > 60%: Check RequestHandlerAvgIdlePercent (PER_BROKER monitoring level). If < 30%, request threads are saturated. Check client batch.size and linger.ms before recommending scaling.

2. KafkaDataLogsDiskUsed > 85% (Standard only): Recommend to the operator that EBS be expanded via aws kafka update-broker-storage (do not execute). Identify high-growth topics via per-topic BytesInPerSec to size the increase. Express clusters use StorageUsed metric instead and storage is fully managed.

3. UnderReplicatedPartitions > 0 (Standard only): Check if a maintenance operation or broker restart is in progress. If URP is decreasing, wait for recovery. Do NOT restart brokers or reassign partitions during URP. Express brokers do not emit this metric — monitor ProduceThrottleTime, FetchThrottleTime, and consumer lag instead.

4. Consumer OffsetLag / MaxOffsetLag increasing: Determine if broker-side (high ProduceTotalTimeMsMean, CPU saturation) or client-side (slow processing, insufficient consumers). Per-partition lag from PER_TOPIC_PER_PARTITION monitoring level helps isolate hot partitions.

5. BytesInPerSec near throughput ceiling: For Standard, check EBS volume type and calculate: BytesInPerSec × ReplicationFactor vs volume throughput limit. For Express, check against the per-broker sustained performance limits in the MSK quotas.

Which Reference Do You Need?

Route to a reference file based on the customer intent. Read the reference in full before answering — do not paraphrase from memory.

For sizing questions (broker count, instance type choice, monthly cost), refer the user to the Amazon MSK best practices — right-size your cluster documentation. Do not size from memory.

Operational Review Workflow

Use this workflow when the user asks for a review, audit, health check, or assessment of an MSK cluster. The routing table above handles ad-hoc troubleshooting; this section produces a consistent, comprehensive report.

Follow the steps in order for each target cluster. Do not skip steps. If a step cannot be completed (e.g. a metric requires a higher monitoring level than the cluster has enabled), record the gap in the report rather than silently omitting the check.

Step 1 — Identify Target Clusters

Ask the user which MSK clusters to review. Accept any of:

• Specific cluster names or ARNs and regions
• "all clusters" in specific regions
• "all MSK clusters in all regions"

If no scope is given, default to all configured account regions. Enumerate clusters with aws kafka list-clusters-v2 per region.

Step 2 — Determine Broker Type Per Cluster

For each cluster:

Copy code
aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>

Read ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType. Standard brokers (kafka.*) and Express brokers (express.*) require different checks in the later steps — some metrics only exist on one type.

Step 3 — Collect Cluster Configuration

For each cluster, gather:

Copy code
aws kafka describe-cluster-v2 --cluster-arn <arn>
aws kafka list-nodes --cluster-arn <arn>
aws kafka get-bootstrap-brokers --cluster-arn <arn>
aws kafka list-cluster-operations-v2 --cluster-arn <arn>   # last 30 days
aws kafka describe-configuration-revision \
     --arn <configuration-arn> --revision <revision>       # if a custom config is applied

Capture:

• Cluster: state, Kafka version, number of broker nodes, AZ distribution (ZoneIds), storage mode (EBS / Tiered), current version.
• Broker: instance type, EBS volume size (Standard), provisioned throughput (if any).
• Encryption: EncryptionInTransit.ClientBroker (TLS / TLS_PLAINTEXT / PLAINTEXT), EncryptionInTransit.InCluster, EncryptionAtRest.DataVolumeKMSKeyId.
• Auth: ClientAuthentication.Sasl.Iam.Enabled, ClientAuthentication.Sasl.Scram.Enabled, ClientAuthentication.Tls.Enabled, ClientAuthentication.Unauthenticated.Enabled.
• Public access: BrokerNodeGroupInfo.ConnectivityInfo.PublicAccess.Type.
• Monitoring level: EnhancedMonitoring (DEFAULT / PER_BROKER / PER_TOPIC_PER_BROKER / PER_TOPIC_PER_PARTITION).
• Logging: LoggingInfo.BrokerLogs (CloudWatch / S3 / Firehose destinations and their Enabled flags).
• Open monitoring: OpenMonitoring.Prometheus.JmxExporter.EnabledInBroker, NodeExporter.EnabledInBroker.
• Recent operations: From list-cluster-operations-v2, note any SECURITY_PATCHING, BROKER_UPDATE, UPDATE_CLUSTER_CONFIGURATION, UPDATE_STORAGE, or UPDATE_MONITORING events in the review window.

Step 4 — Detect Monitoring Level and Gaps

The EnhancedMonitoring value from Step 3 determines which checks are available. At DEFAULT, most per-broker health metrics are still available (CPU, disk, network, partitions, connections, memory, TrafficShaping), but the following checks are not possible without upgrading:

• ReplicationBytesInPerSec / ReplicationBytesOutPerSec (inter-broker replication load)
• RequestHandlerAvgIdlePercent / NetworkProcessorAvgIdlePercent (thread pool saturation)
• BwInAllowanceExceeded / BwOutAllowanceExceeded (detailed bandwidth breaches)
• VolumeQueueLength (EBS I/O queue depth — Standard only)
• VolumeReadBytes / VolumeWriteBytes (EBS throughput utilization — Standard only)
• IAM connection metrics (IAMTooManyConnections)

Record the monitoring level and list any dimensions that will be scored partially or skipped. Recommend upgrading to PER_BROKER if any dimension is degraded by the current level.

Step 5 — Collect CloudWatch Metrics (7-Day Historical)

Namespace: AWS/Kafka. Dimensions: Cluster Name and Broker ID for per-broker metrics; Cluster Name and Consumer Group and Topic for consumer lag; Cluster Name only for cluster-wide metrics.

Use one cloudwatch.GetMetricData batch per cluster where possible. Period: 3600 (1 hour). StartTime: 7 days ago. EndTime: now.

5.1 Cluster-wide (both broker types)

5.2 Per-broker — Standard (kafka.*)

5.3 Per-broker — Express (express.*)

Express brokers do NOT emit: UnderReplicatedPartitions, UnderMinIsrPartitionCount, HeapMemoryAfterGC, TrafficShaping, Volume* metrics, KafkaDataLogsDiskUsed, ProduceMessageConversionsPerSec, FetchMessageConversionsPerSec.

5.4 Consumer lag (both broker types)

Per consumer group (identified from the customer or from the broker logs):

Step 6 — Alarm Coverage

Collect existing alarms:

Copy code
aws cloudwatch describe-alarms --namespace AWS/Kafka

Compare against the 13-alarm recommended set (details in references/monitor-and-alarm.md) and produce a coverage table (present / missing / firing).

Step 7 — Analyze Against Best Practices

Assign a severity to every finding: CRITICAL / HIGH / MEDIUM / LOW / INFO (see Severity Definitions later in this file).

Evaluate across seven dimensions. Some checks are skipped for Express — noted inline.

7.1 Cluster Configuration

• Cluster state is ACTIVE. MAINTENANCE / UPDATING is transient; anything else is a finding.
• Deployed across 3 AZs (Standard: broker count multiple of AZ count).
• Kafka version within N-2 of the latest supported.
• Enhanced monitoring at PER_BROKER or higher — DEFAULT → MEDIUM.
• Storage mode: Tiered storage enabled for topics with long retention (Standard).

7.2 Security

• Encryption in transit: TLS. TLS_PLAINTEXT → MEDIUM (prod: HIGH). PLAINTEXT → CRITICAL.
• Encryption at rest: enabled with KMS. Customer-managed KMS preferred over AWS-managed for regulated workloads.
• Authentication: at least one of IAM / SCRAM / mTLS enabled. Unauthenticated.Enabled=true in production → CRITICAL.
• Public access: SERVICE_PROVIDED_EIPS in production → HIGH.

7.3 Logging & Monitoring

• Broker logs enabled to at least one destination (CloudWatch / S3 / Firehose). All disabled → HIGH.
• Open monitoring (Prometheus JMX + Node exporter) — informational.
• Alarm coverage from Step 6: missing alarms → MEDIUM each; missing critical alarms (Active Controller, Offline Partitions, Disk, CPU) → HIGH.
• Any alarm currently in ALARM state → HIGH (surface in report header).

7.4 Partition Health

• PartitionCount per broker ≤ recommended limit for the broker instance type. Above recommended but below max → MEDIUM. Above max → HIGH (blocks update operations).
• LeaderCount variance across brokers < 10%. 10-25% → MEDIUM. > 25% → HIGH.
• UnderReplicatedPartitions > 0 sustained (Standard only) → HIGH. Transient during a SECURITY_PATCHING / BROKER_UPDATE operation from Step 3 → INFO — do NOT flag.
• UnderMinIsrPartitionCount > 0 (Standard only) → CRITICAL.
• Config-level ISR risk: topics with min.insync.replicas >= replication.factor are a configuration error the CloudWatch metric will not surface. If the user has provided topic configs, flag any such topic as HIGH.

7.5 Compute Health

• CpuUser + CpuSystem avg < 60%. 60-70% → MEDIUM. > 70% → HIGH.
• CPU variance across brokers < 20%. 20-50% → MEDIUM. > 50% → HIGH.
• HeapMemoryAfterGC < 60% (Standard only). 60-80% → MEDIUM. > 80% → HIGH.
• RequestHandlerAvgIdlePercent > 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH.
• NetworkProcessorAvgIdlePercent > 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH.

7.6 Network Health

• TrafficShaping = 0 (Standard only). Any non-zero → HIGH.
• BwInAllowanceExceeded / BwOutAllowanceExceeded = 0 (PER_BROKER). Non-zero → HIGH.
• Throughput variance across brokers < 20%. 20-50% → MEDIUM. > 50% → HIGH.
• Total per-broker throughput < 60% of baseline bandwidth (see references/troubleshoot-performance.md for baseline table). 60-70% → MEDIUM. > 70% → HIGH.
• Express: ProduceThrottleTime / FetchThrottleTime > 0 → HIGH.

7.7 Storage Health

Standard only (Express storage is managed — Express clusters only get the StorageUsed check):

• KafkaDataLogsDiskUsed < 70%. 70-85% → HIGH. > 85% → CRITICAL.
• VolumeQueueLength avg < 1 (PER_BROKER). 1-5 → MEDIUM. > 5 → HIGH.
• EBS auto-scaling configured, or disk headroom > 30% → PASS. No auto-scaling AND disk > 50% → MEDIUM. No auto-scaling AND disk > 70% → HIGH.
• EBS throughput (VolumeReadBytes + VolumeWriteBytes) < 60% of instance baseline (PER_BROKER). 60-70% → MEDIUM. > 70% → HIGH.

Express:

• StorageUsed per broker vs Express per-broker storage quota (see MSK Express quotas). Approaching quota → MEDIUM.

Step 8 — Generate the Report

Generate a separate report artifact per cluster reviewed.

Artifact naming: msk-review-<cluster-name>-<YYYY-MM-DD>.md Example: msk-review-prod-orders-2026-04-29.md

Report structure:

Report Header

Copy code
# MSK Operational Review — <cluster-name>
Account: <account-id> | Region: <region> | Date: <YYYY-MM-DD>
Broker Type: Standard/Express | Instance Type: <type> | Broker Count: <n> | AZs: <n>
Kafka Version: <version> | Monitoring Level: <level>

Executive Summary

• Health: ✅ HEALTHY / ⚠️ WARNINGS / ❌ CRITICAL
• Finding counts by severity
• Top 3 CRITICAL/HIGH items

Configuration Snapshot

| Item | Value | | Cluster state / version | … | | Broker type / instance / count / AZs | … | | Storage | mode, size (Standard), provisioned throughput (if any) | | Encryption | in-transit, in-cluster, at-rest (KMS) | | Authentication | IAM / SCRAM / mTLS / Unauthenticated flags | | Public access | DISABLED / SERVICE_PROVIDED_EIPS | | Monitoring level | DEFAULT / PER_BROKER / PER_TOPIC_PER_BROKER / PER_TOPIC_PER_PARTITION | | Logging | destinations enabled |

Findings by Dimension

For each of the 7 dimensions (7.1-7.7):

| # | Finding | Severity | Current State | Recommendation |

If a dimension was skipped or partial due to monitoring level or broker type, say so explicitly in a note above the table for that dimension.

CloudWatch Metrics (7-Day)

| Metric | Stat | 7-Day Avg | 7-Day Max | Status | Finding |

Alarm Coverage

| # | Recommended Alarm | Metric | Threshold | Priority | Status |

Also list alarms currently in ALARM state with timestamps.

Recent Cluster Operations (Last 30 Days)

From list-cluster-operations-v2 in Step 3:

| Operation Type | Start Time | End Time | State |

Call out any operations that would explain transient metric anomalies.

Priority Matrix

| # | Finding | Severity | Dimension | Effort | Impact |

Sorted by severity.

Next Steps

• Immediate (CRITICAL / HIGH — 7 days)
• Short-term (MEDIUM — 30 days)
• Long-term (LOW — 90 days)

Appendix — Reference Links

• Amazon MSK best practices — Standard brokers
• Amazon MSK best practices — Express brokers
• Amazon MSK — Monitoring an MSK cluster
• Amazon MSK — CloudWatch metrics for Provisioned clusters
• Amazon MSK — Right-size your cluster
• Amazon MSK — Service quotas

Common CLI Recipes

Describe cluster:

Copy code
aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>

List brokers:

Copy code
aws kafka list-nodes --cluster-arn <cluster-arn>

Get bootstrap brokers:

Copy code
aws kafka get-bootstrap-brokers --cluster-arn <cluster-arn>

List recent cluster operations (patching, config updates, storage changes):

Copy code
aws kafka list-cluster-operations-v2 --cluster-arn <cluster-arn>

Expand Standard broker storage — Operator-run (recommend, do not execute):

Copy code
aws kafka update-broker-storage \
  --cluster-arn <cluster-arn> \
  --current-version <cluster-version> \
  --target-broker-ebs-volume-info '[{"KafkaBrokerNodeId": "All", "VolumeSizeGB": <target-size>}]'

Get a CloudWatch metric (example: CpuUser per broker):

Copy code
aws cloudwatch get-metric-statistics \
  --namespace AWS/Kafka \
  --metric-name CpuUser \
  --dimensions Name="Cluster Name",Value="<cluster-name>" Name="Broker ID",Value="<broker-id>" \
  --start-time <start> --end-time <end> --period 300 --statistics Average

Create a cluster configuration (server.properties) — Operator-run (recommend, do not execute):

The --server-properties argument MUST be a real Kafka properties file with one key=value per line, separated by actual newline characters — NOT the literal two-character escape sequence \n. The MSK API accepts the bytes as-is; if you pass "k1=v1\nk2=v2" as a single string with escaped newlines, MSK stores ONE invalid property line and the cluster will fail to apply it.

Recommended pattern: write the properties to a local file with real newlines, then pass it via fileb:// so the CLI uploads the raw bytes verbatim. Verify by reading the revision back with describe-configuration-revision and base64-decoding ServerProperties — you should see one property per line.

Copy code
cat > server.properties <<'EOF'
auto.create.topics.enable=false
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false
num.io.threads=32
num.network.threads=16
log.retention.hours=168
EOF

aws kafka create-configuration \
  --name <config-name> \
  --kafka-versions "3.6.0" \
  --server-properties fileb://server.properties

Common Error Reference

Severity Definitions (for review-style reports)

Additional Resources

• Amazon MSK best practices — Standard brokers
• Amazon MSK best practices — Express brokers
• Amazon MSK — Apache Kafka client best practices
• Amazon MSK — Monitoring an MSK cluster
• Amazon MSK — CloudWatch metrics for Provisioned clusters
• Amazon MSK — Service quotas
• Amazon MSK — Custom MSK configurations