Resolve "OSError: Can't load tokenizer"

You load a tokenizer and get some version of this:

OSError: Can't load tokenizer for 'bert-base-uncased'. If you were trying to
load it from 'https://huggingface.co/models', make sure you don't have a local
directory with the same name. Otherwise, make sure 'bert-base-uncased' is the
correct path to a directory containing all relevant files for a BertTokenizer.

The error message is unusually honest — it tells you the two possibilities right in the text. The problem is that most people read past it. Either you passed a Hub repo id that Transformers could not fetch, or you passed a local folder that is missing the tokenizer files.

Everything else is a special case of those two. This guide splits them cleanly, shows you the one-line check that tells you which you have, and covers the two causes that generate most of these reports: saving the model without the tokenizer, and a local folder shadowing a Hub id.

Quick Answer

Decide whether the name you passed is a Hub repo id or a local folder. For a Hub id: check the spelling, make sure no local folder shares that name, and pass a token if the repo is gated. For a local folder: make sure it contains the tokenizer files — run tokenizer.save_pretrained(dir), not just model.save_pretrained(dir) — and install sentencepiece for T5/Llama-style models.

TL;DR

• Two root causes: a Hub id that can't be fetched, or a local folder missing files.
• The #1 local cause: you saved the model but not the tokenizer.
• The #1 Hub cause: a local folder with the same name shadows the repo.
• SentencePiece models (T5, Llama, XLNet, Gemma) need pip install sentencepiece protobuf.
• Gated/private repo? Pass a token or request access (that path can also 401).
• Corrupted cache? force_download=True or clear ~/.cache/huggingface/hub.

First, Decide: Hub ID or Local Path?

Everything branches from this question, so answer it explicitly:

• Hub repo id — something like bert-base-uncased or google/gemma-3-1b-it. Transformers downloads it (once) and caches it.
• Local folder — a path like ./my-model or /data/checkpoints/run7. Transformers reads the files directly.

A confusing middle case: a bare name that also matches a local folder. If you pass "bert-base-uncased" and a folder named bert-base-uncased exists in your working directory, Transformers treats it as a local path and never touches the Hub. That is the shadowing trap in the next section.

Path A — It's a Local Folder

Cause 1: You saved the model but not the tokenizer

This is the single most common source of the error for fine-tuned models.

model.save_pretrained(dir) writes config.json and the weights. It does not write tokenizer files. So when you later call AutoTokenizer.from_pretrained(dir), there is nothing for it to load.

# ❌ Incomplete — only the model is saved
model.save_pretrained("./my-model")
 
# ... later ...
tok = AutoTokenizer.from_pretrained("./my-model")   # OSError: Can't load tokenizer

The fix is one line — save the tokenizer to the same folder:

# ✅ Complete — save both to the same directory
model.save_pretrained("./my-model")
tokenizer.save_pretrained("./my-model")   # writes tokenizer_config.json, tokenizer.json, ...

• tokenizer.save_pretrained(dir) → emits tokenizer_config.json, tokenizer.json (or the vocab/merges/SentencePiece files), and special_tokens_map.json.
• After this, the folder loads with no error.

What files does the folder actually need?

Different tokenizer families use different files. This is why "just copy vocab.txt" sometimes works and sometimes doesn't:

Confirm what's actually in the folder before guessing:

ls -la ./my-model
# Look for tokenizer_config.json and one of:
# tokenizer.json | vocab.txt | (vocab.json + merges.txt) | tokenizer.model

Cause 2: Missing SentencePiece dependency

SentencePiece tokenizers (T5, Llama, XLNet, Gemma, many multilingual models) need two extra packages. Without them, the files are present but the tokenizer can't be built:

pip install sentencepiece protobuf

Then restart the Python process (a fresh import is required) and load again. If you were forcing the slow tokenizer with use_fast=False, the SentencePiece dependency is mandatory.

Cause 3: You pointed at a file, not a directory

from_pretrained expects a directory, not a single file:

# ❌ a file
AutoTokenizer.from_pretrained("./my-model/tokenizer.json")
# ✅ the directory that contains it
AutoTokenizer.from_pretrained("./my-model")

Path B — It's a Hub Repo ID

Cause 1: A local folder is shadowing the repo

If a folder in your working directory has the same name as the repo id, Transformers loads that folder instead of downloading — and if it lacks tokenizer files, you get the error even though the model exists on the Hub.

import os
print(os.path.isdir("bert-base-uncased"))   # True? that folder is shadowing the Hub

Fixes, cheapest first:

• Run your script from a directory that has no such folder.
• Rename or remove the shadowing folder.
• If you meant the local folder, pass an explicit relative path (./bert-base-uncased) and make sure it has the files.

Cause 2: A typo or wrong revision

Repo ids are case-sensitive and namespaced. Bert-Base-Uncased and bert-base-uncased are different names. Confirm the exact id on the model's Hub page. If you pinned a revision, make sure that branch or tag exists:

AutoTokenizer.from_pretrained("google/gemma-3-1b-it", revision="main")

Cause 3: Gated or private repo without a token

Gated models (many Llama, Gemma, Mistral repos) require you to accept terms and authenticate. Without a token, loading fails — sometimes as this OSError, sometimes as a 401. Authenticate first:

from huggingface_hub import login
login(token="hf_xxx")   # or set the HF_TOKEN environment variable
 
AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

For the auth-specific failure mode, see resolving Hugging Face 401 Unauthorized.

Cause 4: Network, offline mode, or a corrupted cache

• No network on first load: the files aren't cached yet, so the download fails. Load once with connectivity to populate the cache.
• Offline mode: if HF_HUB_OFFLINE=1 is set and the repo isn't cached, it can't fetch. Unset it or pre-cache the model.
• Corrupted / partial cache: an interrupted download leaves broken files. Force a clean re-download:

AutoTokenizer.from_pretrained("bert-base-uncased", force_download=True)

Or clear that model's cache folder under ~/.cache/huggingface/hub and retry.

Verification Steps

1. Load and print the type — success means no OSError:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("./my-model")   # or the Hub id
print(type(tok).__name__)   # e.g. BertTokenizerFast

2. Round-trip a string to prove the vocab loaded:

print(tok.decode(tok("hello world")["input_ids"]))   # -> "[CLS] hello world [SEP]"

3. List the folder (local case) and confirm tokenizer_config.json plus one vocab file are present.
4. Re-run in a clean directory to rule out a shadowing folder.
5. Confirm the dependency for SentencePiece models: python -c "import sentencepiece" runs without error.

Prevention

• Always save the tokenizer next to the model: tokenizer.save_pretrained(dir).
• Never name a working-directory folder after a Hub repo id.
• Pin sentencepiece and protobuf in requirements for T5/Llama-family models.
• Store the exact repo id (with namespace and case) in config, not by memory.
• Pre-cache models in your Docker image build so runtime has no network dependency.
• For gated models, set HF_TOKEN in the environment, not in code.
• Add a smoke test that loads the tokenizer in CI before deploy.

Troubleshooting Matrix

Related Guides

• Resolve Hugging Face 401 Unauthorized
• Fix SentenceTransformer model download failed
• Fix "Token indices sequence length is longer than the specified maximum"
• Resolve CUDA error: device-side assert triggered
• Saving and loading Hugging Face models correctly

External References

• Hugging Face — AutoTokenizer
• Hugging Face — Save and load
• Hugging Face Hub — Authentication
• SentencePiece (Google)

FAQs

Does AutoTokenizer need config.json too? For a local folder, AutoTokenizer reads tokenizer_config.json to pick the class, and often falls back to config.json for the model_type. If both are missing it can't infer the tokenizer, so keep both when you save a checkpoint.

Why does it work on my machine but not in Docker? Usually the model was cached on your machine but the container has no network and an empty cache. Pre-download the model during the image build, or mount a populated cache.

Is this the same as "Repository Not Found"? No. "Repository Not Found" means the id doesn't exist or you lack access. "Can't load tokenizer" means the location was reached (or found locally) but the tokenizer files couldn't be assembled from it.

Can I load a tokenizer straight from a URL? No. from_pretrained takes a Hub repo id or a local directory, not an arbitrary file URL. Download the repo (or clone it) and point at the folder.

I copied vocab.txt but it still fails — why? Because vocab.txt alone isn't enough. You also need tokenizer_config.json so Transformers knows which tokenizer class to build. Save the whole set with save_pretrained rather than hand-copying single files.