Hub Local Cache

This document describes the on-disk layout of the HF Hub local cache. It is intended as a reference for reimplementing the cache system in any language.

Here is a partial list of libraries and applications that use this cache layout. Please open a PR to add your own.

Libraries

Applications

Cache location

The default cache directory is:

~/.cache/huggingface/hub

This can be overridden with environment variables:

• HF_HUB_CACHE - direct path to the cache directory (takes priority)
• HF_HOME - path to the Hugging Face home directory; if set, the cache lives at $HF_HOME/hub

Overview

<CACHE_DIR>/
├── .locks/                                  # Lock files for concurrent download safety
├── blobs/                                   # Shared blob store: Xet files deduplicated across repos
├── models--<org>--<repo>/                   # Cached model repositories
├── datasets--<org>--<repo>/                 # Cached dataset repositories
└── spaces--<org>--<repo>/                   # Cached space repositories

Each downloaded repository gets a single flat folder. Inside each repo folder, files are stored once in a content-addressed blobs/ directory and accessed through snapshots/ symlinks. Named references (branches, tags) are tracked in refs/. Since huggingface_hub 1.32, files downloaded with hf_xet are stored once for the whole cache, in a shared blob store at the cache root. The repo’s blobs/ entry is then a symlink to that file.

Schema

                     ┌──────────────────────────────────────────┐
                     │           Repository folder              │
                     │  models--julien-c--EsperBERTo-small      │
                     └──────────────┬───────────────────────────┘
                                    │
              ┌─────────────┬───────┴───────┬──────────────┐
              │             │               │              │
              v             v               v              v
          ┌───────┐    ┌────────┐    ┌────────────┐   ┌──────────┐
          │ refs/ │    │ blobs/ │    │ snapshots/ │   │.no_exist/│
          └───┬───┘    └────┬───┘    └──────┬─────┘   └────┬─────┘
              │             │               │              │
              │             │               │              │
   "main" contains     Files stored     One folder     Empty marker
   commit hash         by content       per commit     files for
   e.g. "aaaaaa"       hash (SHA-1      hash, e.g.     files known
                       or SHA-256)      aaaaaa/        not to exist
   Resolves a                           bbbbbb/
   branch/tag to                            │
   a snapshot ──────────────────────►  Contains symlinks
                                       to ../../blobs/{hash}

Repository folder naming

Repositories are stored as flat directories at the cache root. The folder name encodes the repo type and repo ID:

{type}s--{repo_id_with_slashes_replaced_by_--}

Rules:

• The repo type is pluralized: models, datasets, spaces
• Forward slashes (/) in the repo ID are replaced with --
• The separator between all parts is --
• Casing is preserved

Examples:

Buckets are not handled by this cache as they are not git-backed. Use the dedicated hf buckets sync command instead.

Inside a repository folder

Every cached repository has the same internal structure:

<repo_folder>/
├── blobs/
├── refs/
├── snapshots/
└── .no_exist/    # may not always be present

blobs/ : content-addressed file storage

The blobs/ directory stores the actual file contents. Each file is named after its file etag on the Hub:

• Git-tracked files: named by their SHA-1 hash (40 hexadecimal characters)
• Git LFS files: named by their SHA-256 hash (64 hexadecimal characters)

This is a flat directory — no subdirectories. Identical files across different revisions are stored only once. An entry is either a regular file or a relative symlink into the shared blob store; readers should follow it.

blobs/
├── 403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd   # SHA-256 (LFS)
├── 7cb18dc9bafbfcf74629a4b760af1b160957a83e                           # SHA-1 (git)
└── d7edf6bd2a681fb0175f7735299831ee1b22b812                           # SHA-1 (git)

refs/ : branch and tag references

The refs/ directory maps human-readable references (branch names, tags, PR numbers) to commit hashes.

Each reference is a plain text file containing a single line: the full commit hash (40 hexadecimal characters).

refs/
├── main            # contains e.g. "bbc77c8132af1cc5cf678da3f1ddf2de43606d48"
├── 2.4.0           # a tag
└── refs/
    └── pr/
        └── 1       # pull request reference

When a file is downloaded using a branch or tag name, the corresponding ref file is created or updated with the latest commit hash.

snapshots/ : revision views

The snapshots/ directory contains one subdirectory per cached revision (commit hash). Each revision directory mirrors the file structure of the repository on the Hub, but files are symlinks pointing into ../../blobs/{hash}.

snapshots/
├── 2439f60ef33a0d46d85da5001d52aeda5b00ce9f/
│   ├── README.md -> ../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812
│   └── pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
└── bbc77c8132af1cc5cf678da3f1ddf2de43606d48/
    ├── README.md -> ../../blobs/7cb18dc9bafbfcf74629a4b760af1b160957a83e
    └── pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd

Key properties:

• Symlinks use relative paths: ../../blobs/{hash}
• If a file is unchanged between two revisions, both symlinks point to the same blob with no data duplication
• Files in subdirectories on the Hub are represented as subdirectories in the snapshot (the full relative path is preserved)

Switching between snapshots is similar to using git checkout in a local git repository.

.no_exist/ : non-existence cache

The .no_exist/ directory tracks files that were requested but do not exist on the Hub. This avoids repeated HTTP requests for optional files.

Structure mirrors snapshots/: one subdirectory per commit hash, containing empty files (not symlinks) named after the missing file.

.no_exist/
└── 2439f60ef33a0d46d85da5001d52aeda5b00ce9f/
    └── config_that_does_not_exist.json    # empty file

Disk usage is negligible since these are only empty marker files.

Shared blob store

Files downloaded through Xet are deduplicated across repositories. Such a file is stored once at the cache root, under blobs/, and each repository that needs it gets a relative symlink instead of a copy:

<CACHE_DIR>/
├── blobs/
│   ├── .huggingface-shared-blobs                 # marker file, contains the layout version ("1")
│   └── 73/                                       # first two hex characters of the Xet hash
│       ├── 73b079dc7bd546aed26d20a82289bd752fc52a2fda13e7f2b551f3af73845b59                       # the file, named by its Xet hash (64 hex characters)
│       ├── 73b079dc7bd546aed26d20a82289bd752fc52a2fda13e7f2b551f3af73845b59.refs                  # manifest: one line per repo blob that references it (path relative to the cache root)
│       └── 73b079dc7bd546aed26d20a82289bd752fc52a2fda13e7f2b551f3af73845b59.lock                  # lock file
│
└── models--HuggingFaceTB--SmolLM2-135M-Instruct/
    └── blobs/
        └── 5af571cbf074e6d21a03528d2330792e532ca608f24ac70a143f6b369968ab8c -> ../../blobs/73/73b079dc7bd546aed26d20a82289bd752fc52a2fda13e7f2b551f3af73845b59

Key properties:

• The repo entry keeps its etag name and becomes a relative symlink: ../../blobs/{xet_hash[:2]}/{xet_hash}
• The snapshots/ layout does not change, so file resolution is unaffected
• The marker file identifies the store: a non-empty blobs/ directory at the cache root without it is never used or modified
• The store requires symlinks and hf_xet. If sharing fails, the file is stored in the repo folder as before. HF_HUB_DISABLE_SHARED_BLOBS=1 disables the store
• Only new downloads go through the store. Files written by older versions or by other clients stay regular files in the repo folder, so a cache can contain both

Implementations that only read the cache need no changes: following symlinks works as before.

Implementations that delete from the cache should treat a repo blob that is a symlink as a reference into the store:

• Delete the symlink in the repo’s blobs/, not the shared file it points to
• Lock <xet_hash>.lock before checking the manifest and deleting
• Delete the shared file and its .refs only if no symlink listed in the manifest still points to it
• If the manifest is missing or can’t be read, keep the shared file

See the huggingface_hub cache guide for deletion behavior and limitations with older clients.

Lock files

Lock files prevent concurrent processes from downloading the same blob simultaneously. They are stored in a .locks/ directory at the cache root (not inside the repo folder):

<CACHE_DIR>/.locks/<repo_folder_name>/<blob_hash>.lock

Example:

<CACHE_DIR>/.locks/models--julien-c--EsperBERTo-small/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd.lock

Full example

~/.cache/huggingface/hub/
├── .locks/
│   └── models--julien-c--EsperBERTo-small/
│       └── 403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd.lock
│
└── models--julien-c--EsperBERTo-small/
    ├── blobs/
    │   ├── [321M]  403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
    │   ├── [ 398]  7cb18dc9bafbfcf74629a4b760af1b160957a83e
    │   └── [1.4K]  d7edf6bd2a681fb0175f7735299831ee1b22b812
    │
    ├── refs/
    │   └── main    # contains "bbc77c8132af1cc5cf678da3f1ddf2de43606d48"
    │
    ├── snapshots/
    │   ├── 2439f60ef33a0d46d85da5001d52aeda5b00ce9f/
    │   │   ├── README.md         -> ../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812
    │   │   └── pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
    │   │
    │   └── bbc77c8132af1cc5cf678da3f1ddf2de43606d48/
    │       ├── README.md         -> ../../blobs/7cb18dc9bafbfcf74629a4b760af1b160957a83e
    │       └── pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
    │
    └── .no_exist/
        └── 2439f60ef33a0d46d85da5001d52aeda5b00ce9f/
            └── optional_config.json    # empty file

Note how pytorch_model.bin points to the same blob in both revisions. The 321 MB file is stored only once on disk.

File resolution logic

To locate a cached file on disk:

1. Resolve the revision to a commit hash

   • If the revision is already a 40-character hex string, use it directly
   • Otherwise, read the file at refs/{revision} to get the commit hash

2. Check the snapshot

   • Look for snapshots/{commit_hash}/{relative_path}
   • If it exists (as a symlink or file), the file is cached. Follow the symlink to get the content

3. Check non-existence

   • Look for .no_exist/{commit_hash}/{relative_path}
   • If it exists, the file is known not to exist on the Hub for this revision

4. Cache miss

   • If neither path exists, the file has not been cached yet

Windows behavior

The cache relies on symbolic links. On Windows systems where symlinks are not available, the cache operates in a degraded mode: actual file copies are placed directly in snapshots/ instead of symlinks. The blobs/ directory and the shared blob store are not used in this mode.

This means the same file content may be duplicated across revisions, increasing disk usage. To enable symlink support on Windows, activate Developer Mode or run as administrator.