Huawei Cloud Dws Spec Planner

Huawei Cloud DWS Data Warehouse Configuration Planning and Sizing

Trigger Words

Chinese triggers: "DWS规划", "DWS选型", "DWS集群规划", "DWS节点规格", "数据仓库迁移DWS", "华为云DWS", "Data Warehouse Service", "DWS sizing", "存算一体", "存算分离", "日增数据量", "DWS扩容", "DWS容量规划"

English triggers: "DWS planning", "DWS sizing", "DWS cluster planning", "DWS node spec", "data warehouse migration to DWS", "Huawei Cloud DWS", "Data Warehouse Service", "coupled storage-compute", "decoupled storage-compute", "daily incremental data", "DWS scale-out", "DWS capacity planning"

Overview

Based on migration or greenfield scenarios, configuration planning is performed for both coupled and decoupled storage-compute architecture modes. The skill outputs one minimum-spec plan that satisfies requirements (minimum compute surplus under constraints), along with storage growth reserve and scale-out cadence recommendations based on daily incremental data. Functional boundary: requirement survey -> architecture mode confirmation -> CPU core calculation -> minimum-spec flavor selection -> hot data storage/cache capacity calculation -> (optional) growth reserve and scale-out cadence -> report output -> Excel generation.

Security rule: Specifications are sourced from Huawei Cloud official document dws_01_00018 (updated 2026-07-31). Scripts must be executed via skill action=exec; direct shell execution is prohibited.

Prerequisites

• Python3 + openpyxl (pip install openpyxl)

Workflow

Phase 1: Requirement Survey

Collect the following key information (8 required + 2 optional):

Source cluster survey checklist: references/source-cluster-survey.md

Phase 2: Architecture Mode Confirmation

Storage calculation rules: references/storage-calculation.md

Phase 3: CPU Core Calculation

• Migration scenario: Total CPU = competitor CPU total (matching configuration); memory ratio follows competitor; when competitor cores not provided, fallback to ceil(data volume/100GB)
• Greenfield scenario: Base cores = ceil(total warehouse data/100GB); compute-intensive x1.3 (memory ratio 1:8), otherwise 1:4
• Both decoupled and coupled use total warehouse data for CPU derivation (v2.5.1); cache ratio only affects storage/cache capacity allocation, not CPU cores
• Constraints: single node >=16vCPU/64GB, node count >=3 (high availability)
• Daily incremental data does not participate in CPU/node derivation; only used for Phase 5 growth reserve

Full CPU calculation logic: references/cpu-sizing-algorithm.md

Phase 4: Node Flavor Planning (Single Minimum-Spec Plan)

Iterate through DWS dedicated flavor catalog (dwsx2/dwsx3/dwsax3 series), satisfying hard constraints (>=16 cores/>=64GB/>=3 nodes), select the combination with minimum compute surplus (nodes + storage). Selection rationale is auto-generated via four-step formula chain: (1) compute derivation -> (2) memory ratio -> (3) concurrency/HA -> (4) storage verification.

Flavor catalog: references/ecs-flavor-catalog.md Selection algorithm: references/single-plan-algorithm.md

Phase 5: Storage Capacity Calculation and Growth Reserve

1. Storage capacity (based on current warehouse volume):

   • Coupled: Effective capacity = total / high watermark (70%); many point queries x(1+index%); per node = ceil(total/nodes, round up to 100GB)
   • Decoupled: Hot data calculated by cache policy -> effective cache = hot data / high watermark (80%) -> same as above
   • Cache policy priority: custom (custom hot data volume) > window (hot window N days) > cost/performance (50%/100%)

2. Growth reserve (optional, requires both daily incremental + retention period):

   • Planning period new data = daily incremental x 30 days/month x retention months; end-of-period total = current + new
   • Scale-out line (90% utilization) capacity -> net growth headroom -> estimated scale-out time point
   • Growth prediction does not change recommended spec; only outputs scale-out recommendations

Growth calculation and Excel template: references/excel-report-template.md

Phase 6: Report Output

Generate structured Markdown report + 5-sheet Excel report:

Common scenarios: references/sizing-scenarios.md

Output Rules

• Output only one plan (minimum spec satisfying requirements, minimum compute surplus); no Plan A/B comparison
• CPU/nodes derived from current warehouse volume; daily incremental does not inflate initial spec (minimum spec philosophy)
• Both decoupled and coupled CPU calculated from total warehouse data (v2.5.1); cache ratio only affects storage/cache capacity allocation
• Missing info handled with defaults but must be annotated (e.g., index space default +30%, cache strategy default performance-optimized 100%)
• Excel report must generate 5 sheets; when daily incremental and retention period not provided, Sheet 4 outputs N/A
• Key decision points must be proactively asked: scenario type, data volume, competitor CPU/memory ratio, compute-intensive, point queries, architecture mode, cache strategy, deployment region

Core Commands

Report Generation

bashCopy code
python scripts/generate_dws_report.py

File naming: DWS_Sizing_Report_<YYYYMMDD>.xlsx

Parameter Confirmation

Notes

1. Production single node >= 16vCPU and >= 64GB (hard requirement); node count >= 3 (high availability)
2. DWS != general ECS: Uses dedicated flavor names (dwsx2/dwsx3/dwsax3 etc.); cannot use MRS aC9/ac7
3. Migration scenario cores: Match competitor cores; no hard >=96 floor; when competitor cores not provided, fallback to ceil(data volume/100GB)
4. CPU-memory ratio: Migration follows competitor; greenfield non-compute-intensive -> 1:4, compute-intensive -> 1:8
5. CPU derived from current warehouse volume: Daily incremental does not inflate initial spec; both decoupled and coupled use total warehouse data (v2.5.1); cache ratio does not affect CPU
6. Coupled high watermark 70% (SSS cloud disk), decoupled high watermark 80% (local cache disk)
7. Index space: Many point queries +30%~50%; default +30% with annotation
8. Cache policy (v2.3.0): cost -> total x50%, performance -> total x100% (default), window -> min(daily incremental x N, total), custom -> given value (<= total)
9. Storage/cache capacity: Configured disk size = actual usable capacity; no replica space reservation needed
10. Source survey caliber (v2.4.0): (1) Distinguish logical volume vs physical volume with replicas (Doris default 3 replicas); (2) Matching cores calculated from source CPU utilization; (3) Memory ratio based on DWS query profile, not copied from source

References

Code Structure

textCopy code
scripts/
└── generate_dws_report.py          # Excel report generation (5 sheets, single plan + growth analysis)
references/
├── acceptance-criteria.md          # Spec output acceptance criteria
├── cpu-sizing-algorithm.md         # CPU core calculation algorithm (migration/greenfield)
├── ecs-flavor-catalog.md           # DWS dedicated flavor catalog (dwsx2/dwsx3/dwsax3 series 1:4/1:8)
├── excel-report-template.md        # Excel report template and report_data data structure
├── iam-policies.md                 # IAM permission requirements (no API permissions needed)
├── single-plan-algorithm.md        # Single-plan minimum-spec selection algorithm and candidate comparison
├── sizing-scenarios.md             # Common planning scenarios (Scenario A-F, with daily incremental growth examples)
├── source-cluster-survey.md        # Source cluster survey checklist (5 groups, 20 items, v2.4.0)
├── storage-calculation.md          # Storage capacity calculation (coupled/decoupled, high watermark, index space)
└── verification-method.md          # Output verification and validation methods

Verification

• Execute python scripts/generate_dws_report.py to generate Excel report; confirm 5 sheets output (Config Recommendation Overview / Resource and Storage Calculation / Selection Decision Rationale / Capacity Growth and Scale-out Recommendations / Decoupled OBS Spec)
• Verify recommended spec satisfies hard constraints: single node >= 16vCPU and >= 64GB, node count >= 3
• Verify decoupled scenario CPU calculated from total warehouse data (same as coupled); cache ratio adjustment does not affect CPU cores
• Confirm daily incremental data does not inflate initial spec (only used for Phase 5 growth reserve)

Best Practices

• DWS uses dedicated flavor names (dwsx2/dwsx3/dwsax3); cannot use MRS aC9/ac7 general-purpose ECS flavors
• Decoupled suitable for >=20TB or low hot data ratio / fast growth scenarios; coupled suitable for <20TB high-performance scenarios
• Cache policy priority: custom > window > cost/performance; select based on business hot data characteristics
• Source survey distinguishes logical volume vs physical volume with replicas (Doris default 3 replicas); matching cores calculated from CPU utilization
• Daily incremental data does not change node spec; only affects scale-out cadence recommendations

Edge Cases

• DWS dedicated flavors: DWS uses dedicated PaaS flavor names (dwsx2/dwsx3/dwsax3 etc.); cannot use general-purpose ECS flavors (e.g., MRS aC9/ac7); must confirm target region DWS flavor availability before sizing
• Architecture switch: Decoupled suitable for >=20TB or low hot data ratio / fast growth; coupled suitable for <20TB high performance. When switching architecture, CPU calculation caliber unchanged (both use total warehouse data); only storage/cache capacity calculation changes (total -> hot data volume)
• Daily incremental boundary: Daily incremental does not participate in CPU/node derivation; only used for Phase 5 growth reserve and scale-out cadence. When daily incremental + retention period not provided, Sheet 4 outputs N/A
• Missing info handling: Missing info handled with defaults but must be annotated (index space default +30%, cache strategy default performance-optimized 100%); silent use of defaults is not allowed
• Source survey caliber distinction: Migration scenario must distinguish logical volume vs physical volume with replicas (e.g., Doris default 3 replicas); matching cores calculated from source CPU utilization; memory ratio based on DWS query profile, not copied from source
• Cache policy priority conflict: custom > window > cost/performance; when multiple strategies provided, use by priority; explain actual effective strategy to user
• Very small data volume: When data <100GB, fallback to ceil(data/100GB) = 1 core; but constrained by single node >=16vCPU hard requirement; actual minimum spec is 16 cores