Large File

Large files: reference, don't load

Scientific files routinely dwarf any context window (90 GB FASTQ, multi-GB HDF5/FITS snapshots, 20 GB+ NetCDF rasters, huge VASP logs). Reading them raw both OOMs and hallucinates — the materials case that consumed 20M+ tokens and failed succeeded in ~1200 tokens with a memory-pointer approach.

Rule: never cat/read a whole data file into your context. Probe it first, work from the returned pointer (schema + sample + key numbers), then read only the specific rows/columns/ranges you need with the real library.

Probe a file

The probe ships beside this SKILL.md. Run it on any data file before opening it:

bashCopy code
python "$XDG_CONFIG_HOME/opencode/skills/large-file/large_file_probe.py" DATA_FILE [--sample N]

It prints one compact JSON pointer on stdout — always tiny, regardless of file size (a 13 MB CSV → ~800 bytes; a 16 MB HDF5 → ~450 bytes).

What you get back

• Tables (CSV/TSV) — column names + inferred dtypes, approximate row count (streamed, constant memory), and a head and tail sample.
• Parquet — schema + row/column/row-group counts, from file metadata only (no column data read).
• HDF5 — the dataset tree with shapes and dtypes (no array data read).
• FITS — HDU list with dimensions and header keys (memmapped headers).
• NetCDF — dimensions and variables with dtypes.
• NDJSON — union of keys, record count, and a sample.
• Genomics (stdlib, gzip-aware — .gz is seen through automatically):
  • FASTQ (.fastq/.fq, incl. .fastq.gz) — read count, read-length min/max/mean over a bounded scan, and sample read ids (never full sequences). A 90 GB FASTQ is counted by streaming, not loaded.
  • FASTA (.fasta/.fa/.fna) — sequence count, total residues, sample ids.
  • VCF (.vcf, incl. .vcf.gz) — variant count, sample names from the #CHROM header, contigs, and a sample of variant rows.
• BAM/CRAM (.bam/.cram) — reference list + header via pysam (header only, no alignment records read).
• GRIB (.grib/.grib2) — variables/coords via cfgrib, or a message sample via pygrib.
• ROOT (.root) — tree/branch listing with entry counts via uproot (metadata only).
• Text / logs — line count and head/tail; scientific logs (VASP OUTCAR, OSZICAR) also get deterministic numeric extraction (e.g. final free energy TOTEN, energy(sigma->0), convergence flag) — the numbers, not the prose.

Binary formats degrade gracefully: if the library (pyarrow/h5py/astropy/ netCDF4/pysam/cfgrib/uproot) isn't installed, the pointer says so with an install hint — it never dumps raw bytes. FASTQ/FASTA/VCF need no library at all.

Then read only what you need

Work from the pointer. When you need actual values, read a bounded slice with the real library — never the whole file:

pythonCopy code
import pandas as pd
df = pd.read_csv("big.csv", nrows=10_000)                 # a bounded window
df = pd.read_csv("big.csv", usecols=["id", "temp_c"])     # only needed columns
import pyarrow.parquet as pq
df = pq.read_table("big.parquet", columns=["val"]).to_pandas()
import h5py
with h5py.File("sim.h5") as h: block = h["density"][0:64, 0:64, :]  # a sub-array

Report which columns/ranges you read, so the analysis stays traceable.