Downloads And Storage

Downloads and Storage

Use this sub-skill to select and operate the Hub's read/download and storage surfaces without confusing repository snapshots, local cache metadata, and bucket object storage.

Trigger Routes

Load this sub-skill when the request mentions any of the following:

• download one Hub file, model, dataset, Space, or kernel repository;
• snapshot a revision with allow/ignore patterns or estimate it with dry run;
• inspect, verify, prune, relocate, or troubleshoot the local Hub cache;
• run offline, recover from an interrupted or incomplete snapshot, or diagnose a cache miss;
• enable, disable, or troubleshoot Xet-backed transfers and chunk storage;
• browse or open Hub files through HfFileSystem, fsspec, or hf:// paths;
• parse a Hub URI or a mount such as hf://datasets/org/data:/mnt:ro;
• browse/download bucket objects or plan a local↔bucket sync;
• reason about trailing-slash semantics for safe remote copy operations.

Read the Focused References

• Read API reference for verified signatures, return values, parameter matrices, URI forms, filesystem methods, and bucket APIs.
• Read workflows for copyable download, offline, cache, fsspec, bucket, copy, and plan/apply recipes.
• Read configuration, cache, and Xet before changing cache paths, offline mode, symlink mode, timeouts, or Xet settings.
• Read troubleshooting for revision/repo-type, access, disk, cache, symlink, Xet, URI, prefix, and traversal failures.
• Run the bundled read-only download environment diagnostic when the effective cache roots, import-time settings, free space, token presence, Xet availability, or cache structure is uncertain.

Keep the Boundaries Clear

This sub-skill owns downloads, read-only filesystem access, local cache operations, bucket reads, and safe movement planning. It covers upload/copy only far enough to choose a safe file-movement surface. Although HfFileSystem exposes write/delete methods, do not execute those mutations from this route; send repository or bucket mutations to the appropriate operations workflow.

• Route repository and bucket creation, commits, branches, uploads, deletes, moves, visibility changes, and other mutations to the hub-operations sibling sub-skill.
• Route exhaustive CLI flag selection, output formats, authentication setup, and shell automation to the cli-and-automation sibling sub-skill.
• Do not use HfFileSystem as the default high-throughput download API; prefer hf_hub_download, snapshot_download, or the corresponding HfApi methods.
• Do not bundle or run credentialed upload helpers or networked installers from this skill.

Choose the Download Surface

1. Use hf_hub_download(repo_id, filename, ...) for exactly one repository file. It returns a file path, normally a pointer inside the shared cache.
2. Use snapshot_download(repo_id, ...) for a coherent repository tree, multiple files, pattern filtering, or parallel file downloads. It returns a snapshot directory.
3. Use local_dir when the caller needs a normal materialized directory they may modify. Otherwise prefer cache_dir/the shared cache and treat returned cache paths as immutable.
4. Set repo_type explicitly for datasets, Spaces, and kernels; model is the default. Pin revision to a full commit hash for reproducible or prepared offline execution.
5. Use dry_run=True to inspect DryRunFileInfo; it may perform remote metadata calls but does not transfer file payloads. It is not an offline inventory mechanism.
6. Use allow_patterns and ignore_patterns only with snapshots. Both apply; ignored files remain excluded after allow filtering. Tune max_workers for concurrent files, not Xet byte-range concurrency.

Before a large or forced transfer, inspect will_download and file_size, run the diagnostic helper, and confirm free space in both the Hub cache and Xet cache locations.

Preserve Cache and Offline Invariants

• local_files_only=True forbids network access for that call and succeeds only from already materialized content. force_download=True asks for a fresh transfer and is not an offline/cache-repair switch; do not combine them.
• HF_HUB_OFFLINE=1 is process-wide and is read when the package is imported. Set it before Python starts. It also makes ordinary HfApi calls fail rather than contact the Hub.
• A cached refs/ entry or trees/*.json record is metadata, not downloaded content. Confirm the selected snapshot files exist.
• If a cached tree listing proves requested files are missing, offline snapshot_download raises IncompleteSnapshotError instead of silently returning the partial tree. Pattern-excluded files do not count as missing.
• Use scan_cache_dir or hf cache ls for inspection. Preview hf cache rm and hf cache prune with --dry-run; deletion is irreversible.
• Never edit a returned shared-cache file. Materialize to local_dir or copy it elsewhere first.

Browse with HfFileSystem and URIs

Use HfFileSystem or the global hffs for fsspec integrations and convenient ls, glob, find, exists, and open. Text mode must be explicit because fsspec open defaults to "rb". Pass refresh=True when stale directory metadata is plausible.

Use canonical hf://[TYPE/]namespace/name[@REVISION][/PATH] URIs. Dataset, Space, kernel, and bucket prefixes are plural and required; a model prefix is optional. Buckets do not support revisions. Encode ordinary branch names that contain /; special refs such as refs/pr/3 are recognized directly. Parse mounts with huggingface_hub.utils.parse_hf_mount, not by splitting on :.

Treat URI parsing as identification, not path-safety authorization. Reject absolute paths, drive/UNC paths, and every .. segment before local materialization or plan application.

Handle Buckets and Movement Safely

Buckets are mutable Xet-backed object storage, not versioned repositories. Use list_bucket_tree/download_bucket_files for Python reads, or hf buckets list, hf cp, and hf sync for CLI workflows.

For one file, use hf cp; for local↔bucket directories, use sync_bucket or hf sync. Generate dry_run=True output or a JSONL plan, review every action and destination, then apply it. Treat delete=True as destructive.

For remote copies, copy_files supports repo→repo, repo→bucket, and bucket→bucket; bucket→repo is unsupported. Server-side copies require compatible storage regions. A trailing slash on the source means “copy contents”; without it, the source directory may be nested at the destination. Verify bucket prefix boundaries so logs does not accidentally match logs-old.

Validate the Outcome

For a no-network regression rehearsal that exercises mocked metadata, a temporary cache, filtered offline recovery, a wrong revision, and the local_files_only/force_download conflict, run the focused case in workflows. Keep the bucket prefix/traversal case there deferred until a reviewed mock plan and authorized bucket integration are available.

• Assert the returned path exists and is the requested file/snapshot.
• For dry runs, inspect filename, commit_hash, local_path, is_cached, will_download, and aggregate only entries where will_download is true.
• For offline snapshots, assert all pattern-selected files exist; catch IncompleteSnapshotError and report its snapshot_path without claiming the snapshot is complete.
• For cache repair, rescan and inspect warnings before deleting anything.
• For sync/copy, inspect source/destination URI types, revisions, prefixes, trailing slashes, regions, and the plan's upload/download/delete actions.
• Mark network-, credential-, private/gated-, and Xet-service integration as deferred unless the caller explicitly authorizes and provides them.