Tablet

Tablet Governance

Investigation guide for tablet health, balance, data skew, bucket optimization, and version-count / compaction-lag issues.

Five root causes account for the vast majority of cases:

• Cause A — data skew from low-cardinality bucket key
• Cause B — too many versions (compaction lag)
• Cause C — error state replicas (VERSION_ERROR, SCHEMA_ERROR)
• Cause D — too many tablets (over-sharding from fine partitions or excess buckets)
• Cause E — disk balance retry loop (metadata state inconsistency)

Metric Taxonomy — Read This First

Before using any metrics, understand the two-layer observability structure:

SQL-based observability

SHOW TABLET FROM <table> — per-tablet summary for a table:

SHOW TABLET <tablet_id> — single tablet detail:

information_schema.be_tablets — per-replica metrics from BE (most detailed):

How to retrieve metrics

Option 1 — SQL (primary method)

sqlCopy code
-- Per-table tablet distribution and health
SHOW TABLET FROM <table_name>;

-- Single tablet detail + detailcmd for replica-level VersionCount
SHOW TABLET <tablet_id>;
-- Run the detailcmd from output:
SHOW PROC '/dbs/<db_id>/<table_id>/<partition_id>';

-- Tablet replica distribution across BEs
ADMIN SHOW REPLICA DISTRIBUTION FROM <table_name>;

-- Deep tablet state per BE (most detailed)
SELECT be_id, tablet_id, num_rowset, num_version, num_segment,
       ROUND(data_size / 1024 / 1024, 1) AS data_size_mb,
       num_row, state
FROM information_schema.be_tablets
WHERE table_id = <table_id>
ORDER BY num_rowset DESC LIMIT 20;

-- Cluster-wide unhealthy tablets
SHOW PROC "/statistic";
SHOW PROC "/statistic/<db_id>";

-- Find tablets with high rowset count (compaction lag indicator)
SELECT be_id, tablet_id, num_rowset, num_segment,
       ROUND(data_size / 1024 / 1024, 1) AS data_size_mb
FROM information_schema.be_tablets
WHERE num_rowset > 100
ORDER BY num_rowset DESC LIMIT 20;

-- Find data skew: tablets with extreme size variance
SELECT be_id, tablet_id,
       ROUND(data_size / 1024 / 1024, 1) AS data_size_mb,
       num_row
FROM information_schema.be_tablets
WHERE table_id = <table_id>
ORDER BY data_size DESC LIMIT 20;

-- Count tablets per BE (distribution balance check)
SELECT be_id, COUNT(*) AS tablet_count,
       SUM(data_size) / 1024 / 1024 / 1024 AS total_gb
FROM information_schema.be_tablets
GROUP BY be_id
ORDER BY tablet_count DESC;

Option 2 — curl BE HTTP port (compaction and rowset state)

bashCopy code
# Compaction status for a specific tablet
curl "http://<be_ip>:8040/api/compaction/show?tablet_id=<tablet_id>"

# All compaction metrics on a BE
curl -s "http://<be_ip>:8040/metrics" | grep -E "compaction_score|num_rowset"

Option 3 — BE HTTP tablet detail

bashCopy code
# Full tablet metadata from BE
curl "http://<be_ip>:8040/api/meta/header?tablet_id=<tablet_id>"

Default HTTP ports: BE = 8040, FE = 8030.

Key rules for interpretation:

1. VersionCount > 500 continuously (from SHOW TABLET) indicates compaction cannot keep up with write rate — this is a compaction problem (see skill 12-compaction), not a tablet governance problem.
2. DataSize variance > 10x across tablets of the same table strongly indicates data skew from a low-cardinality or hot-value bucket key.
3. num_rowset in be_tablets equals VersionCount for non-PK tables; both come from _rs_metas.size(). For PK tables, num_rowset = active rowsets in latest version (usually smaller).
4. LstFailedVersion > 0 and LstFailedVersion != LstSuccessVersion on a replica means that replica has a version gap — it needs repair clone.
5. Total tablet count per cluster should be monitored: > 500,000 tablets creates FE metadata overhead and BE file descriptor pressure.

Tablet Health Reference

Phase 1 — Assess Tablet Health

Step 1.1 — Cluster-wide quick scan

sqlCopy code
-- Check for unhealthy tablets across all databases
SHOW PROC "/statistic";

• UnhealthyTabletNum = 0 everywhere → proceed to performance-focused steps
• UnhealthyTabletNum > 0 → note which DB IDs are affected, go to Step 1.2

Step 1.2 — BE-level tablet scan

sqlCopy code
-- Find tablets with high rowset count (compaction lag signal)
SELECT be_id, COUNT(*) AS tablet_count,
       MAX(num_rowset) AS max_rowsets,
       SUM(CASE WHEN num_rowset > 100 THEN 1 ELSE 0 END) AS tablets_over_100_rowsets,
       SUM(CASE WHEN num_rowset > 500 THEN 1 ELSE 0 END) AS tablets_over_500_rowsets
FROM information_schema.be_tablets
GROUP BY be_id
ORDER BY max_rowsets DESC;

-- Check for severely skewed tablets across all tables
SELECT table_id, be_id,
       MAX(data_size) / 1024 / 1024 AS max_tablet_mb,
       MIN(data_size) / 1024 / 1024 AS min_tablet_mb,
       MAX(data_size) / NULLIF(MIN(data_size), 0) AS size_ratio
FROM information_schema.be_tablets
GROUP BY table_id, be_id
HAVING size_ratio > 10
ORDER BY size_ratio DESC LIMIT 10;

Step 1.3 — Identify issue type

Phase 2 — Identify Problem Tablets

Step 2.1 — Drill into suspect table

sqlCopy code
-- Tablet distribution for the table
SHOW TABLET FROM <table_name>;

-- Replica distribution across BEs
ADMIN SHOW REPLICA DISTRIBUTION FROM <table_name>;

-- Detailed per-replica metrics
SELECT be_id, tablet_id, num_rowset, num_version,
       ROUND(data_size / 1024 / 1024, 1) AS data_size_mb,
       num_row, state
FROM information_schema.be_tablets
WHERE table_id = <table_id>
ORDER BY data_size DESC LIMIT 30;

Step 2.2 — Confirm root cause

Cause A — Data skew

Confirm all match:

• DataSize varies > 10x between tablets of the same table
• num_row also varies proportionally (not a compaction artifact)
• Distribution key has low cardinality or a dominant hot value

→ Go to Phase 3, Cause A

Cause B — Too many versions

Confirm all match:

• num_rowset > 100 in be_tablets for multiple tablets
• VersionCount > 500 from SHOW TABLET <tablet_id>
• Compaction score rising (check skill 12-compaction for compaction-specific diagnosis)

→ Go to Phase 3, Cause B

Cause C — Error state replicas

Confirm all match:

• ErrorStateTabletNum > 0 in SHOW PROC "/statistic"
• SHOW TABLET FROM <table> shows LstFailedVersion > 0 and differs from LstSuccessVersion
• Replica State is not NORMAL

→ Go to Phase 3, Cause C

Cause D — Too many tablets

Confirm all match:

• Total tablet count > 500,000 or tablet count per BE > 50,000
• Many small partitions (< 100MB each) or excess bucket count relative to data size
• BE log shows FD warnings or ulimit -n approaching limit

→ Go to Phase 3, Cause D

Cause E — Disk balance retry loop

Confirm all match:

• SHOW PROC "/cluster_balance/history_tablets" shows same tablet repeatedly failing
• ErrMsg references metadata state or _shutdown_tablets
• BE IO elevated on specific disk without progress

→ Go to Phase 3, Cause E

Phase 3 — Take Action by Cause

Cause A — Data skew (low-cardinality bucket key)

Step A-1: Confirm skew pattern

sqlCopy code
-- Check approximate value distribution in the bucket key column
SELECT <bucket_key_col>, COUNT(*) AS cnt
FROM <table_name>
GROUP BY <bucket_key_col>
ORDER BY cnt DESC LIMIT 20;
-- If top value holds >> 1/bucket_count of total rows → bucket key has hot values

Step A-2: Identify correct bucket key

The bucket key must have high cardinality and uniform distribution. Good candidates:

• User ID, order ID, device ID (UUID-like)
• Composite keys that spread data evenly

Avoid: status, region, date, boolean columns.

Step A-3: Recreate table with better bucket key

sqlCopy code
-- Create new table with better bucket key
CREATE TABLE <table_name>_new
DISTRIBUTED BY HASH(<high_cardinality_col>) BUCKETS <n>
AS SELECT * FROM <table_name>;

-- Swap
ALTER TABLE <table_name> RENAME <table_name>_old;
ALTER TABLE <table_name>_new RENAME <table_name>;

Step A-4: Use StarRocksBuckets CLI for analysis

The standalone StarRocksBuckets utility can analyze existing distributions and generate ALTER TABLE DDL for optimal bucket count and key. Run against the affected table before deciding on the new schema.

Cause B — Too many versions (compaction lag)

Step B-1: Identify affected tablets

sqlCopy code
SELECT be_id, tablet_id, num_rowset,
       ROUND(data_size / 1024 / 1024, 1) AS data_size_mb
FROM information_schema.be_tablets
WHERE num_rowset > 100
ORDER BY num_rowset DESC LIMIT 20;

Step B-2: Confirm compaction type and take action

This is a compaction problem. Use skill 12-compaction for the full diagnosis.

Quick remediation:

bashCopy code
# Manual compaction trigger (cumulative)
curl -XPOST "http://<be_ip>:8040/api/compact?compaction_type=cumulative&tablet_id=<tablet_id>"

# Manual compaction trigger (base — for delete explosion)
curl -XPOST "http://<be_ip>:8040/api/compact?compaction_type=base&tablet_id=<tablet_id>"

sqlCopy code
-- Increase compaction throughput (v3.2+)
UPDATE information_schema.be_configs SET value = 8
WHERE name = 'max_compaction_concurrency';

UPDATE information_schema.be_configs SET value = 2
WHERE name = 'cumulative_compaction_check_interval_seconds';

Step B-3: Reduce import frequency if needed

sqlCopy code
-- Routine Load: batch more rows per interval
ALTER ROUTINE LOAD FOR <job_name>
PROPERTIES (
  "max_batch_interval" = "60",
  "desired_concurrent_number" = "3"
);

Cause C — Error state replicas

Step C-1: Find affected replicas

sqlCopy code
-- Check for version gaps across replicas of a specific tablet
SHOW TABLET <tablet_id>;
-- Run the detailcmd from output
-- Compare Version and LstFailedVersion across replicas

Step C-2: Trigger repair clone

sqlCopy code
-- Mark bad replica to trigger re-replication from healthy replica
ADMIN SET REPLICA STATUS PROPERTIES(
  "tablet_id" = "<tablet_id>",
  "backend_id" = "<bad_be_id>",
  "status" = "bad"
);

Step C-3: Force priority repair

sqlCopy code
ADMIN REPAIR TABLE <table_name>;
ADMIN REPAIR TABLE <table_name> PARTITION (<partition_name>);

Step C-4: Verify

sqlCopy code
-- After repair clone completes, LstFailedVersion should equal LstSuccessVersion
SHOW TABLET <tablet_id>;

Cause D — Too many tablets (over-sharding)

Step D-1: Quantify the problem

sqlCopy code
-- Total tablet count in cluster
SELECT COUNT(*) AS total_tablets FROM information_schema.be_tablets;

-- Tablets per BE
SELECT be_id, COUNT(*) AS tablet_count
FROM information_schema.be_tablets
GROUP BY be_id ORDER BY tablet_count DESC;

-- Find tables with most tablets
SELECT table_id, COUNT(DISTINCT tablet_id) AS tablet_count
FROM information_schema.be_tablets
GROUP BY table_id
ORDER BY tablet_count DESC LIMIT 20;

Step D-2: Check file descriptor pressure

bashCopy code
# Find BE PID
ps aux | grep starrocks_be

# Check current FD usage
ls /proc/<be_pid>/fd | wc -l

# Check FD limit
cat /proc/<be_pid>/limits | grep "open files"

Step D-3: Increase FD limit (immediate relief)

bashCopy code
# In BE startup script or systemd unit
ulimit -n 1000000

# Or in /etc/security/limits.conf
*  soft  nofile  1000000
*  hard  nofile  1000000

Step D-4: Reduce tablet count structurally

sqlCopy code
-- Merge small partitions (use dynamic partitioning at coarser granularity)
-- Example: change daily partitions to monthly
ALTER TABLE <table_name>
SET ("dynamic_partition.time_unit" = "MONTH");

-- Reduce bucket count on over-sharded tables
-- Target: ~1GB per tablet (raw data)
CREATE TABLE <table_name>_new
DISTRIBUTED BY HASH(<col>) BUCKETS <lower_n>
AS SELECT * FROM <table_name>;

-- Drop unused test/backup tables
DROP TABLE <old_table_name>;

-- Implement partition TTL to auto-expire old data
ALTER TABLE <table_name>
SET ("dynamic_partition.history_partition_num" = "90");

Cause E — Disk balance retry loop

Step E-1: Confirm the loop

sqlCopy code
-- Check for same tablet repeatedly failing in balance history
SHOW PROC "/cluster_balance/history_tablets";
-- Look for same TabletId appearing many times with FAILED status

Step E-2: Restart all BE nodes (resets _shutdown_tablets state)

bashCopy code
# Rolling restart: restart BEs one at a time
# Wait for each to rejoin before restarting the next
./bin/stop_be.sh && sleep 5 && ./bin/start_be.sh

After restart, _shutdown_tablets state is cleared and the balance task can proceed cleanly.

Step E-3: Monitor recovery

sqlCopy code
-- Confirm balance tasks completing without repeated failures
SHOW PROC "/cluster_balance/history_tablets";
SHOW PROC "/statistic";

Phase 4 — Verify Recovery

sqlCopy code
-- Primary signal: all tablets healthy
SHOW PROC "/statistic";
-- UnhealthyTabletNum = 0, ErrorStateTabletNum = 0

-- Confirm skew resolved (Cause A)
ADMIN SHOW REPLICA DISTRIBUTION FROM <table_name>;
-- TabletCount should be balanced; RowCount distribution should be even

-- Confirm compaction recovering (Cause B)
SELECT be_id, MAX(num_rowset) AS max_rowsets,
       SUM(CASE WHEN num_rowset > 100 THEN 1 ELSE 0 END) AS tablets_over_100
FROM information_schema.be_tablets
WHERE table_id = <table_id>
GROUP BY be_id;
-- num_rowset should be decreasing

-- Confirm replica health (Cause C)
SHOW TABLET <tablet_id>;
-- LstFailedVersion should equal LstSuccessVersion across all replicas

-- Confirm tablet count reduced (Cause D)
SELECT COUNT(*) AS total_tablets FROM information_schema.be_tablets;

Tablet Governance Config Parameters

Session-level (affects queries on tables, not stored in be.conf):

Common Issues

Related Cases

• case-004-disk-balancing — IO saturation from migration retry loop
• case-014-scan-skew — data skew driven by bucket key with hot value

Causal Chains

Chain 1: Low-Cardinality Bucket Key → Data Skew → Single Tablet Straggler

Copy code
Table created with low-cardinality distribution key
  ↓ observable: SHOW TABLET FROM <table> shows DataSize variance > 10x across tablets
All data for dominant key values routes to one or few tablets
  ↓ observable: information_schema.partitions shows uneven DATA_LENGTH; SHOW PROC '/cluster_balance/tablet_checker' may show oversized tablets
Query scan assigns one instance to the large tablet; parallelism nominally full
  ↓ observable: Profile per-instance Active time in OLAP_SCAN_NODE shows one instance orders of magnitude slower
Query bottlenecked on single slow scan instance; others idle-wait
  ↓ observable: max instance active time >> avg; total query time equals slowest instance

Slow queries despite sufficient cluster resources and parallelism.

Trigger conditions: Distribution key chosen by convenience (status, region) not cardinality; business-domain skewed data.

Break point: Recreate table with high-cardinality distribution key or composite bucket key; increase bucket count.

Chain 2: Inverted Index Pending Tasks → Version Count Rises → Compaction Lag → Scan Slow

Copy code
Inverted index build tasks accumulate due to high write rate or slow index worker
  ↓ observable: BE log inverted index pending task queue depth increasing; index build lag metric rising
Tablet cannot compact rowsets with pending index tasks; version count rises
  ↓ observable: SELECT num_rowset FROM information_schema.be_tablets shows affected tablets with high counts; BE log compaction skipped due to pending index
Compaction falls behind; many unmerged rowsets accumulate
  ↓ observable: Grafana compaction score rising; BE log compaction score warnings
Scan must MERGE many rowsets at query time
  ↓ observable: Profile MERGE time dominant in OLAP_SCAN_NODE; inverted index scan time elevated

Slow queries on tables with inverted indexes under high write load.

Trigger conditions: Inverted index on high-cardinality columns with frequent updates; index worker thread count insufficient.

Break point: Increase inverted index build thread count; temporarily reduce import frequency.

Chain 3: Tablet Count Too High Per BE → FD Limit Hit → Write Fails

Copy code
Table/partition proliferation results in too many tablets per BE
  ↓ observable: SHOW PROC '/cluster_balance/tablet_checker' shows high tablet count per BE; BE log tablet count warning
Each tablet holds open file descriptors for segment, bloom filter, and index files
  ↓ observable: ls /proc/<be_pid>/fd | wc -l approaches ulimit -n value
BE reaches OS fd limit; cannot open new segment files for writes or reads
  ↓ observable: BE log "Too many open files"; ulimit -n shows limit lower than actual FD usage
New segment writes fail; load jobs return IO error from affected BE
  ↓ observable: Load job error: file open failed; TabletCount in Profile very high

Write failures on BEs with extreme tablet counts.

Trigger conditions: Many small partitions without TTL; high bucket count tables; FD limit not tuned beyond OS default (65535).

Break point: Increase ulimit -n for BE process (typically to 1000000); implement partition TTL; reduce bucket count on over-sharded tables.

Resources

• Tablet management documentation