Cpu Saturation

CPU Saturation Triage

Investigation guide for BE or machine CPU saturation. This skill focuses on attribution: which process, which module, and which workload is consuming CPU. After attribution, follow the referenced deep-dive skill for root-cause analysis.

Metric Taxonomy — Read This First

Before using the decision tree, establish what the CPU is doing and which process owns it.

OS-level CPU indicators

How to retrieve:

bashCopy code
# Snapshot CPU breakdown
top -b -n 1 | head -5
# Output: %Cpu(s): 85.2 us,  8.3 sy,  0.0 ni,  3.1 id,  2.1 wa,  0.0 hi,  1.3 si,  0.0 st

# Per-process CPU (10-second sample)
pidstat -u 2 5

# Load average vs CPU count
uptime
# Load average: 64.2, 58.1, 52.3 → compare to: nproc (e.g., 32 CPUs → overloaded)
nproc

BE-level CPU classification metrics (v3.4+)

How to retrieve:

bashCopy code
# Pull all BE CPU task metrics
curl -s "http://<be_host>:<be_http_port>/metrics" | grep "starrocks_be_cpu"

# Loop across all BEs to find the hotspot node
for be in <be1> <be2> <be3>; do
    echo "=== $be ==="
    curl -s "http://$be:8040/metrics" | grep "starrocks_be_cpu_task"
done

FE-level CPU indicators

How to retrieve:

bashCopy code
# FE Java process CPU
top -b -n 1 | grep java

# Per-thread breakdown for FE JVM
top -Hp $(pgrep -f 'StarRocksFE\|fe.pid') | head -20
# Convert thread ID (decimal) to hex → cross-reference with jstack

# FE thread dump
jstack $(cat fe/bin/fe.pid) > /tmp/fe_jstack.log
grep -c "RUNNABLE" /tmp/fe_jstack.log
# High RUNNABLE count relative to vCPUs → planning storm or GC

# GC pressure
grep "GC" fe/log/fe.gc.log | tail -20

Function-level CPU attribution

How to retrieve:

bashCopy code
# Live hot functions (needs root or perf permission)
perf top -p $(pidof starrocks_be) -d 5
# Format: [overhead%]  [symbol_name]  [DSO]
# Focus on the top 5 functions; the prefix identifies the module

# Quick thread snapshot without perf
pstack $(pidof starrocks_be) > /tmp/be_pstack_$(date +%s).log
grep -oE "starrocks::[A-Za-z:]+" /tmp/be_pstack_$(date +%s).log | sort | uniq -c | sort -rn | head -20

Key rules for interpretation:

1. High wa (iowait) is NOT CPU saturation — it means CPU is idle waiting for disk. Diagnose with iostat -x 1, then follow compaction/SKILL.md or import/SKILL.md.
2. starrocks_be_cpu_task_* metrics are available only in v3.4+. On older versions, use perf top or pidstat for function-level attribution.
3. If ALL metrics degrade simultaneously (query + import + compaction), suspect a non-StarRocks process or hypervisor steal (st > 0) rather than a StarRocks-internal cause.
4. BE CPU metrics are per-BE — always identify the hotspot node first before applying cluster-wide changes.
5. FE GC CPU exceeding 30% of elapsed wall time is a separate problem from BE CPU saturation and requires heap/GC tuning, not thread reduction.

CPU Saturation Signal Reference

Triage Decision Tree

Copy code
CPU saturation observed (load average high or top shows 100% on a node)
  │
  ├─ Step 1: Which process owns the CPU?
  │     → see Section 1
  │
  ├─ Process = starrocks_be
  │     └─ Step 2: Which BE module?
  │           → see Section 2
  │           ├─ Queries  → Section 3a + query/SKILL.md
  │           ├─ Import   → Section 3b + import/SKILL.md
  │           ├─ Compaction → Section 3c + compaction/SKILL.md
  │           └─ Background task / unknown → Section 3d
  │
  ├─ Process = java (FE)
  │     → FE Full GC or planning storm → Section 4 + node/SKILL.md
  │
  └─ Process = non-StarRocks (backup agent, kernel, etc.)
        → Section 5: OS-level attribution

1. Which Process Owns the CPU?

bashCopy code
# Snapshot: top CPU consumers by process
top -b -n 1 | head -20

# More detail: per-thread breakdown for a single process
top -Hp $(pidof starrocks_be) | head -30

# Or use pidstat for sustained view
pidstat -u 2 10

2. Which BE Module?

Step 2a: Quick workload inventory

sqlCopy code
-- Any active long-running queries?
SHOW PROC '/current_queries';
-- Look for QueryTime > 30s or MemUsageBytes > 1GB

-- Any active imports?
SELECT LABEL, STATE, PROGRESS, TYPE, LOAD_START_TIME
FROM information_schema.loads
WHERE STATE IN ('LOADING', 'PREPARED', 'PREPARED')
ORDER BY LOAD_START_TIME ASC;

Step 2b: v3.4+ per-task-type CPU metrics

bashCopy code
# CPU breakdown by task type (query / load / compaction / clone / schema change)
curl -s http://<BE_IP>:<BE_HTTP_PORT>/metrics | grep "starrocks_be_cpu"

Key metrics:

• starrocks_be_cpu_task_query — query pipeline CPU
• starrocks_be_cpu_task_load — import write CPU
• starrocks_be_cpu_task_compaction — compaction CPU
• starrocks_be_cpu_task_clone — tablet clone CPU

Step 2c: perf top (function-level attribution)

bashCopy code
# Live hot functions (needs root or perf permission)
perf top -p $(pidof starrocks_be) -d 5

# Or: pstack for a quick thread snapshot without perf
pstack $(pidof starrocks_be) > /tmp/be_pstack_$(date +%s).log

3a. Query-Caused CPU Saturation

Confirm signals:

• starrocks_be_cpu_task_query is the dominant rising metric
• perf top shows starrocks::pipeline:: or starrocks::vectorized:: as hot functions
• SHOW PROC '/current_queries' shows active queries with high ScanRows or MemUsageBytes

Mechanism: Query pipeline threads consume CPU proportional to data volume and operator complexity. Full-table scans (no predicate pushdown), large hash joins (hash table exceeds CPU cache), and high-cardinality aggregations all create sustained CPU pressure. Without resource group limits, a single large query can saturate all cores.

Identify the Culprit SQL

bashCopy code
# Find queries with highest scan rows in audit log (last 1 hour)
python3 scripts/analyze_logs.py "$(date -d '1 hour ago' '+%Y-%m-%d %H:%M:%S')" "$(date '+%Y-%m-%d %H:%M:%S')" "ScanRows" 5 fe.audit.log

# Or find by query execution time
grep 'QueryTime' fe.audit.log | awk -F'QueryTime=' '{print $2}' | cut -d'|' -f1 | sort -rn | head -10

Profile the Query

sqlCopy code
SET is_report_success = true;
-- Re-run the query, then check profile:
-- http://<fe_ip>:8030 -> Queries -> Profile

Common query CPU causes:

→ Deep-dive: query/SKILL.md

3b. Import-Caused CPU Saturation

Confirm signals:

• starrocks_be_cpu_task_load is the dominant elevated metric
• perf top shows starrocks::MemTable::, starrocks::SegmentWriter::, snappy::, or zstd:: as hot functions
• Active imports visible via information_schema.loads

Mechanism: Import flush compresses memtable data before writing to disk segments. Compression (especially zstd) is CPU-intensive. With many concurrent stream loads, flush threads saturate available cores. Query pipeline threads then compete for CPU time slices, causing query latency to rise even without any change in query patterns.

Identify the Import Load

bashCopy code
# Check import thread CPU contribution
curl -s http://<BE_IP>:<BE_HTTP_PORT>/metrics | grep "starrocks_be_cpu_task_load"

# Check active imports
curl -s http://<BE_IP>:<BE_HTTP_PORT>/api/running_load | python3 -m json.tool 2>/dev/null

Common import CPU causes:

→ Deep-dive: import/SKILL.md

3c. Compaction-Caused CPU Saturation

Confirm signals:

• starrocks_be_cpu_task_compaction is the dominant elevated metric
• perf top shows starrocks::Compaction:: or starrocks::SegmentWriter:: as hot functions
• Compaction score was recently high or threads were recently increased to clear a backlog

Mechanism: Compaction threads merge rowsets into larger segments, which involves decompressing source segments, merging sorted data, and recompressing the output. With cumulative_compaction_num_threads_per_disk set too high (e.g., to clear a historical backlog), compaction monopolizes CPU cores, reducing the time slices available to query pipeline threads.

Confirm Compaction Is the Source

bashCopy code
# Check compaction CPU contribution
curl -s http://<BE_IP>:<BE_HTTP_PORT>/metrics | grep "starrocks_be_cpu_task_compaction"

# Check compaction activity
curl -s http://<BE_IP>:<BE_HTTP_PORT>/api/compaction/show | python3 -m json.tool 2>/dev/null

# Check compaction score
SELECT tablet_id, num_rowset, num_segment, data_size
FROM information_schema.be_tablets
ORDER BY num_rowset DESC LIMIT 10;

Common compaction CPU causes:

bashCopy code
# Temporarily reduce compaction threads (dynamic — no restart)
curl -XPOST "http://<BE_IP>:<BE_HTTP_PORT>/api/update_config?cumulative_compaction_num_threads_per_disk=1"

# Or via SQL (v3.2+, applies to all BEs)
# UPDATE information_schema.be_configs SET value = 1
# WHERE name = 'cumulative_compaction_num_threads_per_disk';

→ Deep-dive: compaction/SKILL.md

3d. Unknown Background Task

Confirm signals:

• perf top shows hot functions that don't match query/import/compaction prefixes
• SHOW PROC '/current_queries' shows no long-running queries
• information_schema.loads shows no active imports

Mechanism: Schema changes, tablet clone (rebalance), and memory allocator contention (tcmalloc) can cause sustained CPU usage without appearing in the query or import workload metrics. Clone operations copy tablet data between BEs during rebalancing; schema changes rebuild column segments. Both generate significant IO and CPU workload.

If perf top shows hot functions that don't match query/import/compaction prefixes:

bashCopy code
# Check schema change tasks
SHOW ALTER TABLE COLUMN;

# Check clone tasks
SHOW PROC '/cluster_balance';

# Check routine load tasks
SHOW ROUTINE LOAD;

# Memory allocator contention (tcmalloc)
curl -s http://<BE_IP>:<BE_HTTP_PORT>/memz | head -30

4. FE (Java) CPU Saturation

Confirm signals:

• top shows java process (FE) in top CPU consumers
• jstack shows many RUNNABLE threads in CBO optimizer or statistics code
• fe.gc.log shows frequent Full GC events

Mechanism: FE Java process CPU saturation has two distinct causes: (1) query planning storm where many concurrent complex queries overwhelm CBO planner threads, and (2) Full GC where Java's garbage collector consumes CPU cleaning heap. Both appear as high java process CPU but have different jstack signatures.

bashCopy code
# Identify hot Java threads
top -Hp $(pgrep -f 'StarRocksFE\|fe.pid') | head -20
# Convert thread ID to hex, then cross-reference jstack

jstack $(cat fe/bin/fe.pid) > /tmp/fe_jstack.log
# Look for threads in RUNNABLE state consuming CPU

# Check if GC is the cause
grep "GC" fe/log/fe.gc.log | tail -20

→ Deep-dive: node/SKILL.md — Section 3 (FE GC) / Section 4 (thread pool)

5. Non-StarRocks Process CPU

Confirm signals:

• ps aux --sort=-%cpu shows a non-StarRocks process in the top CPU consumers
• st (steal) > 0 in top (VM environments)
• All StarRocks metrics (query + import + compaction) degrade simultaneously

Mechanism: Co-located processes (backup agents, Spark jobs, monitoring scanners) compete for CPU with StarRocks. In VM environments, hypervisor steal (st) directly reduces the CPU available to StarRocks without appearing as a StarRocks-internal issue. The distinguishing signal is simultaneous degradation of ALL metrics — if only one metric category is affected, the cause is StarRocks-internal.

bashCopy code
# Identify the process
ps aux --sort=-%cpu | head -20

# Check if it's a StarRocks-adjacent tool
ls /proc/$(pgrep <process_name>)/exe -l  # What binary is it?

# Check cgroup CPU usage if StarRocks is in a cgroup
cat /sys/fs/cgroup/cpu/starrocks/cpu.stat 2>/dev/null

Common non-StarRocks culprits:

Mitigation: Use cgroup or systemd resource limits to cap non-StarRocks CPU:

bashCopy code
# Cap a process to 50% of 2 cores
systemctl set-property <service> CPUQuota=100%

# Or use cpulimit for an ad-hoc process
cpulimit -p $(pgrep rsync) -l 20

CPU Throttling Config Parameters

Applying Dynamic Parameters

bashCopy code
# Per BE (v2.5+): update compaction threads dynamically
curl -XPOST "http://<be_ip>:<be_http_port>/api/update_config?cumulative_compaction_num_threads_per_disk=1"
curl -XPOST "http://<be_ip>:<be_http_port>/api/update_config?max_compaction_concurrency=4"

# Verify
curl "http://<be_ip>:<be_http_port>/varz" | grep compaction_num_threads

sqlCopy code
-- v3.2+: apply to all BEs at once
UPDATE information_schema.be_configs SET value = 4
WHERE name = 'max_compaction_concurrency';

-- Verify
SELECT * FROM information_schema.be_configs WHERE name = 'max_compaction_concurrency';

Static parameters (flush_thread_num_per_store, clone_worker_count, brpc_num_threads) require be.conf change and BE restart.

Causal Chains

Chain 1: Heavy Scan Query → BE CPU Saturation → Query Feedback Loop

Copy code
Large full-table-scan or multi-join query submitted without resource group
  ↓ observable: SHOW PROC '/current_queries' shows query with ScanRows >> expected; no LIMIT
Pipeline executor threads saturate available CPU cores
  ↓ observable: top -Hp <be_pid> shows many threads at 100%; perf top shows vectorized:: hot
Other queries and imports compete for CPU; scheduling latency rises
  ↓ observable: concurrent simple queries start taking seconds; import write latency rises
New queries slow → users retry → more load submitted → CPU stays saturated (feedback loop)
  ↓ observable: SHOW PROC '/current_queries' count growing; no queries completing

Cluster appears frozen; all operations slow; restarting query does not help until root cause is killed.

Trigger conditions: No resource group limits; ad-hoc analytics on production cluster; missing WHERE filter causes full-table scan.

Break point: KILL QUERY <query_id> for the offending query; set query_mem_limit and resource groups to prevent recurrence.

Chain 2: High-Frequency Import → Memtable Flush Compression → CPU Saturation

Copy code
High-throughput import with zstd compression on memtable flush
  ↓ observable: perf top shows snappy:: or zstd:: hot; starrocks_be_cpu_task_load elevated
Flush thread pool saturates CPU; query pipeline gets fewer CPU slices
  ↓ observable: query latency increases; SHOW PROC '/current_queries' shows elevated QueryTime
Queries appear slow despite no change in query patterns
  ↓ observable: profile shows similar execution plan but higher wall time; CPU wa low (not IO)

Queries degrade without any query-side change; root cause is import workload added to the same BEs.

Trigger conditions: Large-batch import with high compression ratio (zstd/snappy) while serving online queries; flush_thread_num_per_store too high relative to available cores.

Break point: Reduce flush_thread_num_per_store; switch to lz4 compression for memtable (compression_type = "LZ4" in table properties); add resource group CPU cap for import tasks.

Chain 3: Compaction Thread Over-Provisioned → CPU Contention → Query Slowdown

Copy code
Compaction thread count set high to clear a historical backlog
  ↓ observable: starrocks_be_cpu_task_compaction elevated; Grafana shows compaction bytes/sec high
Compaction threads consume a sustained fraction of CPU cores
  ↓ observable: top shows BE near 100%; perf top shows Compaction:: or SegmentWriter:: functions
Query pipeline gets fewer CPU time slices; scan and join throughput drops
  ↓ observable: query latency rises proportionally to compaction CPU share; profile shows longer operator times
Compaction-query CPU contention sustained until backlog cleared
  ↓ observable: query latency normalizes once compaction score drops below threshold

Queries slow for hours after a compaction tuning change; operators suspect query regression but root cause is compaction.

Trigger conditions: cumulative_compaction_num_threads_per_disk increased to catch up on a backlog without considering query workload; no CPU cap on compaction tasks.

Break point: curl -XPOST http://<BE>/api/update_config?cumulative_compaction_num_threads_per_disk=1 to reduce compaction pressure dynamically without restart.

Chain 4: Non-StarRocks Process Steals CPU → All Operations Slow

Copy code
Scheduled backup / monitoring agent launches and consumes CPU (no resource limit set)
  ↓ observable: top shows non-StarRocks process in top 3 by CPU; ps aux --sort=-%cpu confirms
StarRocks BE and FE receive fewer CPU cycles from OS scheduler
  ↓ observable: CPU steal (%st in top) > 0 if in VM; or BE CPU user% drops while load average stays high
Queries slow; imports slow; compaction falls behind
  ↓ observable: ALL metrics degrade simultaneously — if only queries slow, suspect query cause
Non-StarRocks process finishes or is killed → all StarRocks metrics recover
  ↓ observable: latency returns to baseline without any StarRocks config change

Simultaneous degradation of all metrics (query + import + compaction) at the same time is the key signal for an OS-level cause rather than a StarRocks-level cause.

Trigger conditions: Co-located workloads on StarRocks nodes; no cgroup or systemd resource limits for non-StarRocks processes; scheduled jobs (backup, log rotation, monitoring scan) running during peak hours.

Break point: kill or cpulimit the offending process; long-term: enforce cgroup CPU quotas for non-StarRocks processes, or move StarRocks to dedicated hosts.

Common Issues

Related Skills

• query/SKILL.md — query profiling and scan optimization
• import/SKILL.md — import pipeline and flush tuning
• compaction/SKILL.md — compaction thread and score management
• node/SKILL.md — FE GC, BE OOM, deadlock
• resource-isolation/SKILL.md — resource groups and query governance

Related Cases

• case-014-scan-skew — query-caused CPU spike from scan skew and MERGE-heavy tablets
• case-008-be-oom — BE OOM triggered by CPU-saturating compaction during large query burst

Resources

• StarRocks Performance Tuning
• Linux perf basics