Alterlab Zarr

Zarr Python

Overview

Zarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray. The guidance targets zarr-python 3.x (current 3.4.0), which writes the Zarr v3 format by default.

When to Use This Skill

Use this skill when the user wants to:

• Create, open, or write chunked and compressed N-D arrays (zarr.create_array, groups, attributes).
• Choose chunk shapes, sharding, and codecs (Zstandard, Blosc, Gzip) for an access pattern.
• Put arrays on S3/GCS or other object stores and read slices remotely (consolidated metadata, sharding).
• Convert HDF5/NumPy/NetCDF data to Zarr, or migrate Zarr v2-era code (compressor=, synchronizer=, create_dataset) to v3.

Does NOT Trigger

Quick Start

Installation

bashCopy code
uv pip install zarr

Zarr v3 (zarr>=3); zarr 3.2+ requires Python 3.12+ and NumPy 2 (the 3.1.x line still supports Python 3.11). For cloud storage support, install the matching fsspec backend:

bashCopy code
uv pip install s3fs   # For S3
uv pip install gcsfs  # For Google Cloud Storage

Basic Array Creation

pythonCopy code
import zarr
import numpy as np

# Create a 2D array with chunking and compression
z = zarr.create_array(
    store="data/my_array.zarr",
    shape=(10000, 10000),
    chunks=(1000, 1000),
    dtype="f4"
)

# Write data using NumPy-style indexing
z[:, :] = np.random.random((10000, 10000))

# Read data
data = z[0:100, 0:100]  # Returns NumPy array

Core Workflow

1. Create or open an array/group, picking a store appropriate to the environment (local, in-memory, ZIP, S3/GCS).
2. Choose chunking aligned to your access pattern (aim for 1-10 MB chunks; rows-first → chunks span columns, and vice versa). This is the single biggest performance lever.
3. Pick compression via compressors= based on workload — Zstandard (the default), Blosc+LZ4 (fast), Gzip (max ratio); compressors=None to disable. Pass Blosc options as strings (shuffle="shuffle", cname="zstd"); the BloscShuffle/BloscCname enums are deprecated since 3.1.
4. Read/write with NumPy-style indexing; resize/append as data grows.
5. Scale out with Dask (lazy, out-of-core, parallel) or label with Xarray for climate/geospatial data.
6. For cloud and many-array stores, consolidate metadata and consider sharding to cut object/file count. Consolidated metadata is a zarr-python convention, not yet part of the v3 spec (zarr warns), so other readers may ignore it.

pythonCopy code
# Minimal end-to-end
import zarr, numpy as np
z = zarr.create_array(store="data/my_array.zarr", shape=(10000, 10000),
                      chunks=(1000, 1000), dtype="f4")
z[:, :] = np.random.random((10000, 10000))
sub = z[0:100, 0:100]            # returns a NumPy array

Routing — where to look

Additional Resources

• Official Documentation: https://zarr.readthedocs.io/
• Zarr Specifications: https://zarr-specs.readthedocs.io/
• GitHub Repository: https://github.com/zarr-developers/zarr-python
• Related: Xarray (labeled arrays), Dask (parallel computing), NumCodecs (compression codecs)

Part of the AlterLab Academic Skills suite.