Nw Data Architecture Patterns

Organization

nWave-ai

nw-data-architecture-patterns

Data architecture patterns (warehouse, lake, lakehouse, mesh), ETL/ELT pipelines, streaming architectures, scaling strategies, and schema design patterns

Overview

Publisher	nWave-ai
Repository	`nWave`
Skill name	`nw-data-architecture-patterns`
Stars	616
Forks	65
Bundled files	Instructions only
License	MIT
Links	Skill source Repository Homepage

Markdown instructions
A SKILL.md file the model loads on demand, so it only costs tokens when a request actually matches.
Works with any LLM
AI skills are plain Markdown, not provider-specific code, so this works with GPT, Claude, Gemini, Grok, or a local model.
Self-contained
Everything the model needs lives in the instructions — no extra files to sync.
Open source
Published by nWave-ai on GitHub. Read the source before you install it.

Installation

Install the Nw Data Architecture Patterns AI skill in TypingMind to use it with any LLM, or drop it into another agent that reads SKILL.md.

Install in TypingMind

TypingMind installs a skill straight from its GitHub folder — it reads SKILL.md, bundles the resource files, and stores the result locally.

Open the app and go to Plugins → Skills.
Choose "Install from GitHub".
Paste the skill folder URL below and confirm.
https://github.com/nWave-ai/nWave/tree/main/nWave/skills/nw-data-architecture-patterns
Enable the skill in any chat where you want it available.

Plugins → Skills → Add skill → From GitHub URL, then paste the folder URL and press Continue.

Install in another agent

Any agent that reads the Agent Skills format can use this skill — copy the folder into that agent's skills directory.

Claude Code — .claude/skills

git clone --depth 1 https://github.com/nWave-ai/nWave.git /tmp/nWave
mkdir -p .claude/skills
cp -r /tmp/nWave/nWave/skills/nw-data-architecture-patterns .claude/skills/nw-data-architecture-patterns

Restart Claude Code after copying so it picks up the new skill.

Use it in TypingMind

Enable Nw Data Architecture Patterns in any TypingMind chat and the model takes it from there. Its name and description sit in the system prompt, and the moment a request matches, the model loads the full instructions itself — you never invoke it by hand, and it costs no tokens until it is actually used.

The model loads Nw Data Architecture Patterns on its own as soon as a request matches it.

Works with any AI model

AI skills are plain Markdown instructions rather than provider-specific code, so Nw Data Architecture Patterns is not tied to the model it was written for. Install it once in TypingMind and use it with GPT-5, Claude, Gemini, Grok, DeepSeek, Mistral, Llama, or a local model you run yourself — all on your own API keys.

Loaded only when it is needed
The system prompt carries just the name and description. The instructions are fetched on the first matching request, so an idle skill costs nothing.
Switch models mid-chat
Because the skill is instructions rather than code, changing model does not break it — the next model reads the same SKILL.md.

Skill instructions

This is the SKILL.md content the model loads. Read it before installing — a skill is instructions your model will follow.

Data Architecture Patterns

Architecture Selection Decision Tree

Structured only -> Data Warehouse | Mixed + SQL analytics -> Data Lakehouse | Mixed + ML-primary -> Data Lake | Large org + autonomous domains -> Data Mesh

Data Warehouse

Schema: structured, schema-on-write | Data: tables, rows, columns | Governance: centralized | Query: SQL analytics, BI | Architecture: centralized single source of truth

Schema Patterns

Star Schema: Central fact table (measures) surrounded by denormalized dimension tables. Best for BI dashboards, standard reporting.

Snowflake Schema: Normalized dimensions (dimensions reference other dimensions). Reduces storage, increases JOIN complexity. Best when storage cost matters more than query speed.

Kimball vs Inmon

Kimball (Bottom-Up): Build data marts first, integrate later | Star schema, business-process driven | Faster initial delivery | Best for quick wins, department-level analytics

Inmon (Top-Down): Build enterprise DW first, derive data marts | Normalized 3NF enterprise model | Higher upfront effort | Best for large enterprises needing single source of truth

Technology: Snowflake | Amazon Redshift | Google BigQuery | Azure Synapse Analytics

Data Lake

Schema-on-read, flexible | All formats (structured, semi-structured, unstructured) | Raw data in native format | Query via Athena, Spark SQL, PySpark, Pandas | Risk: "data swamp" without governance

Organization

Zones: raw (landing, original format) -> curated (cleaned, validated) -> processed (transformed for use cases) -> archive (cold storage)

Anti-Patterns

No metadata catalog -> undiscoverable data
No access controls -> security/compliance risk
No data quality checks -> garbage in/out
No retention policy -> unbounded cost growth

Technology: S3 + Athena/Glue | Azure Data Lake Storage + Synapse | HDFS + Hive

Data Lakehouse

Combines warehouse reliability with lake flexibility | Schema enforcement on write with evolution support | ACID transactions on lake storage | Supports both BI/SQL and ML/data science workloads

Medallion Architecture (Bronze / Silver / Gold)

Bronze: Raw data as-is, append-only for auditability, partitioned by ingestion date, schema-on-read Silver: Quality rules (null checks, range validation, referential integrity) | Deduplication on business keys | Schema enforced | SCD applied Gold: Business-level aggregations | Dimensional models (star/snowflake) | Pre-computed metrics/KPIs | Optimized for BI/reporting

Technology: Databricks (Delta Lake) | Apache Iceberg | Apache Hudi

Data Mesh

Core Principles (Martin Fowler)

Domain-oriented ownership: Data owned by domain teams, not central
Data as a product: Each domain publishes discoverable, trustworthy, self-describing data products
Self-serve data platform: Infrastructure team provides platform for domain teams
Federated computational governance: Global standards with domain autonomy

ETL vs ELT Pipeline Design

ETL (Extract-Transform-Load)

Transform before loading via dedicated engine (Informatica, Talend, SSIS). Best for complex transforms, constrained targets, regulatory requirements. Scaling limited by transform engine.

ELT (Extract-Load-Transform)

Load raw first, transform using target compute (dbt, Snowflake SQL, BigQuery SQL). Best for cloud DWs with elastic compute, preserving raw data. Scales with target system.

Pipeline Design Principles

Idempotency: Re-running produces same result (use MERGE/upsert, not INSERT)
Incremental processing: Process only new/changed data (watermarks, CDC)
Schema evolution: Handle added/removed columns gracefully (schema registry)
Data quality gates: Validate between stages (null rates, row counts, value ranges)
Observability: Log metrics (rows processed, duration, errors, freshness)

Orchestration

Apache Airflow: DAG-based, Python-native, wide adoption | Prefect: modern, dynamic workflows | Dagster: software-defined assets

Streaming Architecture

Apache Kafka

Distributed event streaming platform. Concepts: topics, partitions, consumer groups, offsets. At-least-once delivery (exactly-once with transactions). Use as event bus, message broker, stream storage.

Apache Flink

Stateful stream processing engine. Concepts: DataStreams, windows (tumbling, sliding, session), state management. Exactly-once with checkpointing. Common pattern: Sources -> Kafka (durable event buffer) -> Flink (stateful compute) -> Sinks.

Architecture Selection

Streaming: real-time dashboards, fraud detection, IoT, event-driven | Batch: overnight reporting, historical analysis, ML training | Lambda: parallel batch + stream (complex, prefer Kappa) | Kappa: stream-only, reprocess from Kafka log (simpler)

Scaling Strategies

Vertical (Scale Up)

Add CPU/RAM/storage to existing server | Simpler ops, no app changes | Hard limit: largest hardware | Use first for moderate growth

Horizontal (Scale Out)

Read Replicas: Replicate to read-only copies | Route reads to replicas, writes to primary | Trade-off: replication lag (eventual consistency) | Use for read-heavy workloads

Partitioning (Single Server): Range (date, alphabetical) | List (region, category) | Hash (even distribution) | Benefits: query pruning, maintenance (drop old partitions)

Sharding (Multiple Servers): Distribute data across DB instances by shard key | Strategies: range-based, hash-based, directory-based, geographic

Shard Key Selection (most impactful decision):

High cardinality for even distribution
Even access frequency to avoid hot shards
Query alignment: most queries target single shard
Avoid monotonically increasing keys (hot spots)

Challenges: Cross-shard queries need scatter-gather | Distributed transactions (2PC) complex/slow | Resharding expensive | App complexity increases

Scaling Decision Guide

Not exceeding single server -> optimize queries/indexes first | Read-heavy -> add read replicas | Write-heavy + partitionable -> partition then shard | Write-heavy + not partitionable -> write-optimized DBs (Cassandra, DynamoDB)

Normalization vs Denormalization

Practical approach: Start normalized for transactional tables | Add denormalized/materialized views for reporting | Denormalize selectively based on measured performance | Document decisions and rationale

Frequently asked questions

What does the Nw Data Architecture Patterns AI skill do?

Data architecture patterns (warehouse, lake, lakehouse, mesh), ETL/ELT pipelines, streaming architectures, scaling strategies, and schema design patterns

Why use Nw Data Architecture Patterns on TypingMind?

Because you install it once and use it with any model. Nw Data Architecture Patterns is plain Markdown rather than provider-specific code, so the same skill runs on GPT-5, Claude, Gemini, Grok, or a local model — and you can switch model mid-chat without it breaking. TypingMind runs on your own API keys, so you pay providers directly instead of a per-seat subscription, and your skills and chats stay in your own storage.

How do I install Nw Data Architecture Patterns in TypingMind?

Open Plugins → Skills → Install from GitHub in TypingMind and paste https://github.com/nWave-ai/nWave/tree/main/nWave/skills/nw-data-architecture-patterns. TypingMind reads its SKILL.md and installs it as a skill you can enable per chat.

Which AI models can use Nw Data Architecture Patterns?

Any model you connect in TypingMind. AI skills are plain Markdown instructions rather than provider-specific code, so GPT, Claude, Gemini, Grok, and local models can all load this skill when a request matches it.

How many AI models can I use with Nw Data Architecture Patterns?

As many as you like. As long as a model supports skills, you can use Nw Data Architecture Patterns with it — GPT, Claude, Gemini, Grok, DeepSeek, Mistral, Llama and more — all on TypingMind with your own API keys.

Is the Nw Data Architecture Patterns AI skill free?

Yes. It is published on GitHub by nWave-ai under the MIT license. You only pay your own AI provider for the tokens you use.

What are AI skills?

An AI skill is a reusable instruction bundle that teaches an AI model how to do one specific task. It follows the open Agent Skills format: a SKILL.md file with a name and description, plus any scripts, templates or reference files the model may need. The model reads the instructions only when your request matches the skill, so an installed skill costs nothing until it is used.

How are AI skills different from plugins or MCP servers?

A plugin or MCP server gives a model new tools to call — code that runs somewhere and returns a result. An AI skill gives the model knowledge and process instead: how to approach a task, which steps to follow, what good output looks like. Skills are plain Markdown, so they need no server, no API key and no runtime, and they work with any model.

View all

Caveman Stats

JuliusBrussee

CommunityPopular

Show recorded output and cache-read token usage and mode attribution for the current Claude Code session, or locate the host's native usage report. Trigger: /caveman-stats.

Ci Metrics

Query PyTorch CI, GitHub Actions, HUD, Grafana, and infrastructure metrics. Use when users ask about CI duration, job failures, queue times, workflow trends, runner health, dashboard data, or PyTorch…

Notcrawl

Notion archive: search, sync freshness, pages/databases, Markdown exports, SQL counts, and Notcrawl repo work.

Openclaw Test Heap Leaks

openclaw

OrganizationPopular

Investigate OpenClaw pnpm test memory growth, Vitest OOMs, RSS spikes, and heap snapshot deltas.

Overview

Installation

Install in TypingMind

Install in another agent

Use it in TypingMind

Works with any AI model

Skill instructions

Data Architecture Patterns

Architecture Selection Decision Tree

Data Warehouse

Schema Patterns

Kimball vs Inmon

Data Lake

Organization

Anti-Patterns

Data Lakehouse

Medallion Architecture (Bronze / Silver / Gold)

Data Mesh

Core Principles (Martin Fowler)

ETL vs ELT Pipeline Design

ETL (Extract-Transform-Load)

ELT (Extract-Load-Transform)

Pipeline Design Principles

Orchestration

Streaming Architecture

Apache Kafka

Apache Flink

Architecture Selection

Scaling Strategies

Vertical (Scale Up)

Horizontal (Scale Out)

Scaling Decision Guide

Normalization vs Denormalization

Frequently asked questions

Related AI Skills

Set up your own AI workspace now