Databricks Iceberg

Apache Iceberg on Databricks

Databricks provides multiple ways to work with Apache Iceberg: native managed Iceberg tables, UniForm for Delta-to-Iceberg interoperability, and the Iceberg REST Catalog (IRC) for external engine access.

Critical Rules (always follow)

• MUST use Unity Catalog — all Iceberg features require UC-enabled workspaces
• MUST NOT install an Iceberg library into Databricks Runtime (DBR includes built-in Iceberg support; adding a library causes version conflicts)
• MUST NOT set write.metadata.path or write.metadata.previous-versions-max — Databricks manages metadata locations automatically; overriding causes corruption
• MUST determine which Iceberg pattern fits the use case before writing code — see the When to Use section below
• MUST know that both PARTITIONED BY and CLUSTER BY produce the same Iceberg metadata for external engines — UC maintains an Iceberg partition spec with partition fields corresponding to the clustering keys, so external engines reading via IRC see a partitioned Iceberg table (not Hive-style, but proper Iceberg partition fields) and can prune on those fields; internally UC uses those fields as liquid clustering keys; the only differences between the two syntaxes are: (1) PARTITIONED BY is standard Iceberg DDL (any engine can create the table), while CLUSTER BY is DBR-only DDL; (2) PARTITIONED BY auto-handles DV/row-tracking properties, while CLUSTER BY requires manual TBLPROPERTIES on v2
• MUST NOT use expression-based partition transforms (bucket(), years(), months(), days(), hours()) with PARTITIONED BY on managed Iceberg tables — only plain column references are supported; expression transforms cause errors
• MUST disable deletion vectors and row tracking when using CLUSTER BY on Iceberg v2 tables — set 'delta.enableDeletionVectors' = false and 'delta.enableRowTracking' = false in TBLPROPERTIES (Iceberg v3 handles this automatically; PARTITIONED BY handles this automatically on both v2 and v3)

Key Concepts

Quick Start

Create a Managed Iceberg Table

sqlCopy code
-- No clustering
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
AS SELECT * FROM raw_events;

-- PARTITIONED BY (recommended for cross-platform): standard Iceberg syntax, works on EMR/OSS Spark/Trino/Flink
-- auto-disables DVs and row tracking — no TBLPROPERTIES needed on v2 or v3
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
PARTITIONED BY (event_date)
AS SELECT * FROM raw_events;

-- CLUSTER BY on Iceberg v2 (DBR-only syntax): must manually disable DVs and row tracking
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
TBLPROPERTIES (
  'delta.enableDeletionVectors' = false,
  'delta.enableRowTracking' = false
)
CLUSTER BY (event_date)
AS SELECT * FROM raw_events;

-- CLUSTER BY on Iceberg v3 (DBR-only syntax): no TBLPROPERTIES needed
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
TBLPROPERTIES ('format-version' = '3')
CLUSTER BY (event_date)
AS SELECT * FROM raw_events;

Enable UniForm on an Existing Delta Table

sqlCopy code
ALTER TABLE my_catalog.my_schema.customers
SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Read/Write Capability Matrix

Reference Files

When to Use

• Creating a new Iceberg table → references/1-managed-iceberg-tables.md
• Making an existing Delta table readable as Iceberg → references/2-uniform-and-compatibility.md
• Making a streaming table or MV readable as Iceberg → references/2-uniform-and-compatibility.md (Compatibility Mode section)
• Choosing between Managed Iceberg vs UniForm vs Compatibility Mode → decision table in references/2-uniform-and-compatibility.md
• Exposing Databricks tables to external engines via REST API → references/3-iceberg-rest-catalog.md
• Integrating Databricks with Snowflake (either direction) → references/4-snowflake-interop.md
• Connecting PyIceberg, OSS Spark, Flink, EMR, or Kafka → references/5-external-engine-interop.md

Common Issues

Related Skills

• databricks-unity-catalog — catalog/schema management, governance, system tables
• databricks-pipelines — SDP pipelines (streaming tables, materialized views with Compatibility Mode)
• databricks-python-sdk — Python SDK and REST API for Databricks operations
• databricks-dbsql — SQL warehouse features, query patterns

Resources

• Iceberg Overview — main hub for Iceberg on Databricks
• UniForm — Delta Universal Format
• Iceberg REST Catalog — IRC endpoint and external engine access
• Compatibility Mode — UniForm for streaming tables and MVs
• Iceberg v3 — next-gen format features
• Foreign Tables — reading external catalog data