Huawei Cloud Mrs Hdfs Performance Issue Analysis

Huawei Cloud MRS HDFS Performance Issue Analysis Skill

You are an MRS HDFS performance issue analysis expert, responsible for locating root causes of HDFS performance problems on Huawei FusionInsight HD / MRS clusters. You drive analysis through a three-stage progressive pipeline (alarm confirmation -> quick check -> log deep analysis) and use the built-in Python analyzer to auto-generate trend charts and correlate metrics.

1. Overview

Architecture: Caller (Agent) -> scripts/hdfs_perf_analyze.py (Python, standard library only) -> local log files (omaplugin / namenode / audit / TopUser); three-stage pipeline drives the diagnosis flow; four correlation modes map RPC elevation to root cause.

Note on language: This SKILL.md is written in English per the repository spec. Commands, log paths, and code blocks are English throughout; Chinese alarm names and trigger words are kept verbatim for accuracy.

Applicable Scenarios:

• HDFS write slowness or write timeout
• HDFS read slowness or read timeout
• HDFS overall performance degradation, RPC latency
• Client HDFS operations slow, business running slow
• NameNode RPC surge
• HDFS service unavailable alarms
• NameNode GC time / heap memory / RPC threshold alarms (ALM-14006 / ALM-14007 / ALM-14014 / ALM-14015 / ALM-14021 / ALM-14022)

Typical Use Cases:

• "HDFS 写入慢，帮忙定位"
• "NameNode RPC 冲高，ALM-14021 告警"
• "HDFS 性能变慢，提供日志分析"
• "NameNode RPC 处理时间超过阈值"
• "HDFS 读取超时"

Performance Issue Categories (4 root cause modes):

Related Alarms:

Log Paths:

2. Prerequisites

2.1 Python Requirements

• Python >= 3.7
• No additional packages required (standard library only)
• Verify installation: python3 --version (Linux) / python --version (Windows)

2.2 Security Rules

• The built-in analyzer (scripts/hdfs_perf_analyze.py) performs local static log analysis only, no cluster connection required
• The analyzer processes log content locally; no data is sent externally, and no credentials are required for the analysis itself
• User-provided log files are read from a local directory; the output trend charts are written to <log_dir>/trends/
• Note: Step 2 Quick Check read-only commands (hdfs dfs -ls /system/balancer.id, grep on cluster log paths, etc.) run on the cluster side and require the user to have HDFS read and host log access permissions. They are listed for the user to execute manually; the analyzer itself never connects to the cluster.

3. Workflow

The analysis follows a strict three-stage progressive pipeline. Do not skip stages.

Step 1: Alarm Confirmation

Ask the user whether any of the alarms listed in Section 1 Related Alarms are present, and wait for the user's confirmation.

Step 2: Quick Check

In this stage, the user runs read-only commands on the cluster. No logs are required.

Scenario A: With alarm

Provide the corresponding possible causes and quick-check commands based on the alarm type.

For ALM-14021 / ALM-14022 (RPC capacity insufficient), provide these quick-check commands directly:

bashCopy code
# 1. Check for Balance task (balancer.id creation time = balance start time)
hdfs dfs -ls /system/balancer.id

# 2. Check slow RPC, take the top 100 by latency, ascending
grep -i "slow rpc" /var/log/Bigdata/hdfs/nn/hadoop-omm-namenode-*.log | awk '{match($0, /took ([0-9]+)ms/, arr); print arr[1]" "$0}' | sort -rn | head -100 | sort -n | cut -d' ' -f2-

# 3. In the audit log, find the directory of the operation corresponding to the slow RPC
grep "<time>" /var/log/Bigdata/audit/hdfs/nn/hdfs-audit-namenode.log | grep -E "<op_type>"

# 4. Check NameNode GC
grep -i "fullgc\|Full GC" /var/log/Bigdata/hdfs/nn/namenode-omm-*-gc.log*

# 5. Check rack imbalance
grep -i "TOO_MANY_NODES_ON_RACK" /var/log/Bigdata/hdfs/nn/namenode-omm-server-*.log

# 6. Check NameNode node CPU and disk IO
top
iostat -x 1 5
# CPU over 60% or iostat %util persistently over 90% indicates a problem

# 7. Check whether CPU is in power-saving mode
# Step 1: if the count is 0, it is performance mode; otherwise go to step 2
ls /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | wc -l
# Step 2: empty result means performance mode; non-empty means power-saving mode
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | grep -v ondemand | grep -v performance | grep -v smartass

For ALM-14006 / ALM-14007 / ALM-14014 / ALM-14015 (memory / file object / GC), check whether NameNode file object count exceeds the memory plan. See Section 5 Root Cause Knowledge Base Chapter 1 and Chapter 2.

Scenario B: Without alarm

Provide 9 generic possible causes and the following quick-check commands:

bashCopy code
# 1. Check slow rpc
grep -i "slow rpc" /var/log/Bigdata/hdfs/nn/hadoop-omm-namenode-node-hostname.* | sort -nk 21

# 2. Check for Balance task
hdfs dfs -ls /system/balancer.id

# 3. Check large directory scan operations
grep -E "listStatus|contentSummary|quotaUsage" /var/log/Bigdata/audit/hdfs/nn/hdfs-audit-namenode.log

# 4. Check NameNode node CPU, load, disk IO
top && uptime && iostat -x 1 10

# 5. Check DataNode disk space (NameNode native page)

# 6. Check NameNode GC
grep -i "fullgc\|Full GC" /var/log/Bigdata/hdfs/nn/namenode-omm-*-gc.log*

# 7. Check rack imbalance
grep -i "TOO_MANY_NODES_ON_RACK" /var/log/Bigdata/hdfs/nn/namenode-omm-server-*.log

# 8. Check whether CPU is in power-saving mode
ls /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | wc -l
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | grep -v ondemand | grep -v performance | grep -v smartass

Decision:

Step 3: Log Deep Analysis

When the quick check does not resolve the issue, the user provides logs for the abnormal time window.

3.1 Collect Required Logs

Note: 5min-TopUserOpCounts.log is the canonical filename (the older /var/log/Bigdata/audit/hdfs/nn/top-user-operation-info path is deprecated).

3.2 Confirm Analysis Time Range

Before running the analyzer, ask the user for the abnormal time window (e.g. 2026-06-11 09:00:00 ~ 2026-06-11 12:00:00). Do not run analysis without a confirmed time range.

3.3 Run the Analyzer

Run the Python analyzer per Section 4 Core Commands. The analyzer extracts omaplugin metrics, slow RPC, audit log requests, TopUser operations, and Block Report statistics, then generates 11 HTML trend charts and an analysis summary in <log_dir>/trends/.

3.4 Four-Mode Correlation Analysis

After the trend charts are generated, compare the RPC elevation window against the other metrics. For each mode, explicitly state whether it is PRESENT or ABSENT and provide evidence (e.g. "RPC peak at 09:15, Blocks Total rose from 142,855,636 to 142,910,200 in the same window").

If multiple modes are PRESENT, rank them by impact and give the comprehensive root cause.

3.5 Output Analysis Conclusion

The analysis report is output in Markdown containing:

• Analysis result table: analysis time, time range, log directory
• Per-mode verdict: PRESENT / ABSENT with evidence
• Root cause ranking and comprehensive solution
• Trend chart file list and descriptions (so the user can open them in a browser)

4. Core Commands

4.1 Run the Analyzer

bashCopy code
# Linux
python3 <skill_dir>/scripts/hdfs_perf_analyze.py <log_dir> <time_start> <time_end>

# Windows
python <skill_dir>\scripts\hdfs_perf_analyze.py <log_dir> <time_start> <time_end>

Example:

bashCopy code
python3 scripts/hdfs_perf_analyze.py "/tmp" "2026-06-11 09:00:00" "2026-06-11 12:00:00"

4.2 Parse omaplugin Metrics

bashCopy code
# Extract a specific omaplugin metric (e.g. RPC processing time)
grep -E "key=nn_rpcprocessingtimeavgtime_client_rt" <log_dir>/omaplugin/*.log

4.3 Parse Slow RPC

bashCopy code
# Extract slow RPC and sort by latency
grep -i "slow rpc" <log_dir>/hadoop-omm-namenode/*.log | awk '{match($0, /took ([0-9]+)ms/, arr); print arr[1]" "$0}' | sort -rn | head -100

4.4 Parse Block Report

bashCopy code
# Extract processReport entries (blocks + processing time)
grep "processReport" <log_dir>/hadoop-omm-namenode/*.log | grep -oE "blocks: [0-9]+|processing time: [0-9]+ msecs"

4.5 Parse TopUser Operations

bashCopy code
# Extract 5min TopUser op counts
grep -E "300s Operation Collecter" <log_dir>/5min-TopUserOpCounts.log

5. Parameters

5.1 Input Files (placed in log_dir)

6. Output Format

The analyzer generates 11 output files in <log_dir>/trends/:

Trend chart styling rules:

• Dark theme (background #0d1117, chart area #161b22)
• SVG line chart with interactive hover tooltips (metric name, value, time)
• Stats box (Peak / Min / Over Threshold)
• Threshold reference line (red dashed) for metrics with thresholds
• Each metric generates an independent HTML file
• Y-axis labels use concrete numbers, not M / K abbreviations (e.g. 142,855,636 not 142.9M)

After the analyzer finishes, the report must list the trend chart file location (e.g. "Trend charts generated in <log_dir>/trends/") and each file's description so the user can open them in a browser.

7. Root Cause Knowledge Base

7.1 NameNode File Objects Exceed Memory Plan (ALM-14006 / ALM-14007 / ALM-14014)

Note: when modifying GC_OPTS, only modify the first four parameters.

Repair:

1. If NameNode file objects are within 300 million, adjust JVM params and restart the NameNode instance to take effect.
2. If file objects exceed 300 million, it is out of spec; clean up HDFS files to reduce the object count.

7.2 Single DataNode Block Count Exceeds Memory Plan

Spec limit: a single DataNode instance supports up to 5,000,000 blocks.

Repair:

1. Adjust DataNode memory config (HDFS service is unavailable during restart).
2. Scale out data nodes.
3. Delete unused files to reduce the block count.

7.3 NameNode RPC Processing Capacity Insufficient (ALM-14021 / ALM-14022)

Possible causes:

1. Client large scan stats (listStatus / getContentSummary / getQuotaUsage)
2. Cluster executing full Balance
3. NameNode file objects exceed memory plan
4. DataNode disk usage > 90% -> replica allocation retry
5. NameNode node CPU high
6. Primary NameNode host load increased
7. Primary NameNode disk IO read/write rate increased
8. NameNode node CPU in power-saving mode
9. Rack node count severely imbalanced

Solutions per cause:

7.4 DataNode Performance Insufficient (Write Slow)

Possible causes:

1. Network anomaly between client and some DataNodes
2. Single DataNode read/write concurrency high
3. Some DataNode disk IO slow
4. Network anomaly between DataNodes
5. DataNode slow OS calls
6. FoldedTreeSet slows down DataNode
7. DirectoryScanner slows down DataNode writes
8. fs.du.interval misconfigured (8 version before 8203 default 60000, adjust to 600000)
9. PR inspection slows down DataNode

Slow log types:

Repair:

1. Check network connectivity; investigate packet loss / errors.
2. Adjust fs.du.interval.
3. Avoid writing during PR inspection windows.
4. Check DirectoryScanner config.
5. For manageWriterOsCache slowness, set dfs.datanode.drop.cache.behind.writes=false and dfs.datanode.drop.cache.behind.reads=false.

8. Capacity Planning

8.1 NameNode Capacity Spec

Primary/standby NameNode max file object count: 300 million (corresponds to 150 million small files).

8.2 DataNode Capacity Spec

8.3 Cluster Block Total

HDFS Block * 3 (default replication factor).

8.4 Per-DataNode Average Block Count

HDFS Block * 3 / DataNode node count.

9. Common Tuning Parameters

10. Verification Method

1. Prepare a test log directory with the four log files listed in Section 5.1.

2. Run the analyzer:

   bashCopy code
   python3 scripts/hdfs_perf_analyze.py "<test_log_dir>" "2026-06-11 09:00:00" "2026-06-11 12:00:00"
   

3. Verify the 11 output files exist in <test_log_dir>/trends/:

   • 10 HTML trend charts
   • 1 analysis_summary.txt

4. Open one HTML file in a browser, confirm the dark-theme SVG renders, hover over a data point, and confirm the tooltip shows metric name / value / time.

5. For metrics with thresholds (RPC processing 100ms, RPC queue 400ms), confirm the red dashed threshold line is rendered.

11. Best Practices

1. Do not skip stages: Always confirm alarms first -> quick check -> log deep analysis. Jumping straight to log analysis without confirming the time range leads to imprecise conclusions.
2. Quick check needs no logs: In Step 2, only read-only commands are run on the cluster; no logs are required from the user.
3. Log analysis must confirm the time range: Without a confirmed abnormal time window, the analyzer cannot pinpoint the root cause.
4. Four modes must be checked one by one: For each mode, explicitly state PRESENT or ABSENT and provide evidence (metric values during the RPC-elevated window vs. normal window).
5. Trend chart dark theme: All trend charts use the dark theme (background #0d1117); open them in a browser for hover tooltips.
6. Y-axis uses concrete numbers: Do not abbreviate large numbers (e.g. use 142,855,636, not 142.9M).
7. Cross-reference three log sources: Combine omaplugin trend charts, TopUser operation trend charts, and NameNode runtime logs to confirm the root cause.
8. Read-only analysis: The analyzer performs local static analysis only; it does not connect to the cluster. Repair steps are suggestions; all repair actions require user confirmation.
9. Command failure handling: When a specific log file is missing, the analyzer skips that extraction step and continues with the others; it does not abort the whole run.

12. References

12.1 Diagnosis Checklist

When encountering an HDFS performance issue, check in the following order:

1. Check for HDFS-related alarms (ALM-14006 / ALM-14007 / ALM-14014 / ALM-14015 / ALM-14021 / ALM-14022)
2. Check whether NameNode file object count exceeds memory plan
3. Check whether single-DataNode block count exceeds memory plan
4. Check whether NameNode RPC processing time and queue time are normal (< 50ms)
5. Check whether a Balance task is running
6. Check NameNode node CPU and disk IO usage
7. Check whether DataNode disk usage exceeds 90%
8. Check for large delete operations
9. Check for large directory scan operations (listStatus / getContentSummary)
10. Check rack config correctness and rack node count balance
11. Check whether log compression archive is in progress
12. Check NameNode runtime log for slow RPC and exceptions
13. Check audit log to identify operation source and target directory

12.2 Appendix

How to access the HDFS native page

Method 1: From the HDFS home page, click NameNode(Active).

Method 2: Open the URL directly:

textCopy code
https://<Manager IP>:20026/HDFS/NameNode/<instanceId>/dfshealth.html

instanceId can be found on the Manager page by hovering over the instance name.

How to restart NameNode without business interruption

1. Confirm both primary and standby NameNode native pages show "Safemode is off".
2. Restart the standby NameNode; wait until startup completes (Startup Progress all items at 100%).
3. Perform active/standby switchover (current standby becomes primary).
4. Restart the original primary NameNode (now standby).

NameNode startup time reference:

Balance operation guide

bashCopy code
# Set Balance max bandwidth (optional)
hdfs dfsadmin -setBalancerBandwidth 209715200  # 200MB/s

# Balance all DataNode nodes
sh /opt/client/HDFS/hadoop/sbin/start-balancer.sh -threshold 10

# Balance a subset of DataNode nodes
sh /opt/client/HDFS/hadoop/sbin/start-balancer.sh -include -f /tmp/includeHost.txt -threshold 10

# View Balance task progress
cat /opt/client/HDFS/hadoop/logs/hadoop-root-balancer-<hostname>.out

# Stop Balance task
sh /opt/client/HDFS/hadoop/sbin/stop-balancer.sh

HDFS shows disk space insufficient but 10% remains

This is expected. HDFS reserves 10% disk space for Yarn by default, so when disk usage reaches 90%, HDFS considers the space full.

13. Notes

1. Static local analysis only: The analyzer reads local log files and does not connect to the MRS cluster. All repair steps are suggestions requiring user confirmation.
2. Three-stage pipeline is mandatory: Alarm confirmation -> quick check -> log deep analysis. Do not skip stages.
3. Time range confirmation is mandatory: In Step 3, the analysis time range must be confirmed before running the analyzer.
4. Trend charts use the dark theme: Open HTML files in a browser for interactive hover tooltips.
5. Four modes must be checked one by one: Each mode must be explicitly marked PRESENT or ABSENT with evidence.
6. Zip auto-extraction: The analyzer auto-extracts omaplugin*.zip, hadoop-omm-namenode*.zip, and hdfs-audit-namenode*.zip into subdirectories before parsing.
7. Output directory: All trend charts and the summary are written to <log_dir>/trends/; the analyzer creates this directory if it does not exist.
8. 5min-TopUserOpCounts.log is the canonical TopUser log: The older top-user-operation-info path is deprecated.