Fix "Token indices sequence length is longer than the specified maximum"

You tokenize some text, and Hugging Face prints this in red:

Token indices sequence length is longer than the specified maximum sequence
length for this model (2048 > 512). Running this sequence through the model
will result in indexing errors.

Here is the thing most guides get wrong: this is not the error. Your code did not crash. The tokenizer finished and handed you every token — all 2048 of them. The message is a warning that the tokenizer prints because it can see the crash coming. The real failure happens a few lines later, when those 2048 tokens hit a model that only has room for 512.

Get that mental model right and the fix is obvious. Miss it, and you end up "fixing" the warning by silencing it — right before your model throws IndexError: index out of range in self or, on GPU, the dreaded CUDA error: device-side assert triggered.

This guide covers the fix, the subtle model_max_length sentinel trap that makes truncation=True silently do nothing, and how to keep the whole document when truncating would lose information.

Quick Answer

The message is a warning: your text tokenized to more tokens than the model allows (e.g. 2048 > 512), but the tokenizer still returned all of them. Feeding them to the model then crashes. Fix it by tokenizing with truncation=True and an explicit max_length. If truncation seems ignored, your tokenizer's model_max_length is the huge sentinel value — pass max_length yourself.

TL;DR

• It's a warning, not the error. The tokenizer returns every token; the crash comes later.
• Downstream crash: IndexError: index out of range in self (CPU) or CUDA error: device-side assert triggered (GPU).
• The fix: tokenizer(text, truncation=True, max_length=512).
• The trap: some tokenizers have no model_max_length — it's int(1e30), so truncation=True alone truncates to nothing. Pass max_length explicitly.
• You can't raise the ceiling. Positional embeddings are fixed at training time.
• Keep all the text: sliding window (return_overflowing_tokens + stride) or a long-context model (Longformer, BigBird, LED).

The Warning Is a Preview of the Crash

This is the single most important idea in this article. The tokenizer's job is to turn text into token ids. It does that job successfully even when the result is too long — it just warns you first. The model's job is to look up an embedding for each token and each position. A BERT-style model trained with 512 positions has an embedding table with exactly 512 position rows. Ask it for position 1500 and there is no row — that is the index-out-of-range crash.

Reproduce It

A minimal, complete example that shows the warning and the crash it predicts:

import torch
from transformers import AutoTokenizer, AutoModel
 
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")  # max length 512
model = AutoModel.from_pretrained("bert-base-uncased")
 
long_text = "hello world " * 1000  # ~2000 tokens
 
# 1) The WARNING is printed here — but this line succeeds.
enc = tokenizer(long_text, return_tensors="pt")
print("token count:", enc["input_ids"].shape[1])   # e.g. 2002 — all kept
 
# 2) The ERROR is raised here, when the model sees position 512+.
with torch.no_grad():
    model(**enc)   # IndexError: index out of range in self

• Line 8 prints the warning but returns successfully — proof it is not the error.
• The print confirms the tokenizer kept all ~2000 tokens.
• Line 13 is where it actually breaks. On CPU you get IndexError; move the same code to CUDA and you get device-side assert triggered instead.

Common Causes

Symptoms

• A red Token indices sequence length is longer… line, with numbers like (2048 > 512).
• IndexError: index out of range in self in the traceback, pointing at an nn.Embedding lookup.
• On GPU: RuntimeError: CUDA error: device-side assert triggered, often with Assertion srcIndex < srcSelectDimSize failed.
• Training that works on short samples but crashes on the first long one.
• Embeddings or predictions that look wrong because text was silently cut off.

Root Cause

Two fixed numbers collide:

1. The model's maximum positions. BERT and RoBERTa were trained with 512 positional embeddings. GPT-2 has 1024. These are baked into the weights.
2. Your token count. Whatever your text tokenizes to.

When (2) exceeds (1), the model has no position embedding to look up, so the lookup indexes past the end of the table. The tokenizer cannot fix this for you — it does not know whether you want to truncate, chunk, or switch models — so it warns and hands back everything.

The model_max_length sentinel trap

This is the part almost every other article skips, and it is why people say "truncation=True doesn't work for me."

Hugging Face only sets a real model_max_length for tokenizers it has a hardcoded entry for. For many models — especially custom or community ones — the value defaults to a sentinel:

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("some-custom/model")
print(tok.model_max_length)   # 1000000000000000019884624838656  (int(1e30))

truncation=True truncates to max_length, and max_length defaults to model_max_length. If that is int(1e30), there is effectively no limit, so your over-long sequence sails straight through. You will also sometimes see:

Asking to truncate to max_length but no maximum length is provided and the
model has no predefined maximum length. Default to no truncation.

That message is the tokenizer telling you, in its own words, that truncation=True did nothing.

Step-by-step Solution

Step 1: Truncate with an explicit max_length (the real fix)

Never rely on truncation=True alone. Always pair it with a number:

enc = tokenizer(
    long_text,
    truncation=True,
    max_length=512,        # explicit — never trust the default
    return_tensors="pt",
)
print(enc["input_ids"].shape[1])   # 512 — safe for the model

• truncation=True → drop tokens past the limit.
• max_length=512 → the limit itself; set it to your model's real maximum.
• Result: a tensor the model can consume without crashing.

Step 2: Batch tokenization with padding

For real workloads you tokenize batches. Combine truncation with padding so every sequence in the batch is a safe, uniform length:

enc = tokenizer(
    list_of_texts,
    truncation=True,
    max_length=512,
    padding="max_length",   # or padding=True for dynamic padding
    return_tensors="pt",
)

• padding="max_length" → pad every sequence up to max_length (fixed shape).
• padding=True → pad only to the longest sequence in the batch (faster, less memory).

Step 3: Verify the sentinel isn't biting you

Before trusting truncation, confirm the tokenizer knows its own limit:

print(tokenizer.model_max_length)
 
# If it prints the int(1e30) sentinel, pin it once:
if tokenizer.model_max_length > 100_000:
    tokenizer.model_max_length = 512

This guard means a future truncation=True call without an explicit max_length will still behave. (Passing max_length explicitly, as in Step 1, is the more robust habit — do both.)

Keeping the Whole Document

Truncation throws away everything after the limit. When the tail matters — long contracts, articles, transcripts — you need a strategy that preserves it.

Option A: Sliding window with return_overflowing_tokens

The tokenizer can split a long text into overlapping windows for you. This is the correct, built-in way to chunk — no manual string slicing:

enc = tokenizer(
    long_text,
    truncation=True,
    max_length=512,
    stride=128,                       # overlap between windows
    return_overflowing_tokens=True,
    return_tensors="pt",
)
print(enc["input_ids"].shape)   # e.g. torch.Size([5, 512]) — 5 windows

• return_overflowing_tokens=True → emit multiple windows instead of dropping the overflow.
• stride=128 → each window repeats the last 128 tokens of the previous one, so context isn't lost at the seams.
• You then run all windows through the model and aggregate (mean-pool embeddings, max over class scores, etc.).

Option B: Use a long-context model

If you frequently need the full document as one representation, switch to a model built for long inputs instead of fighting a 512-token one.

Positional embeddings in these models cover thousands of positions, so the warning never fires for normal documents. The trade-off is higher memory and slower inference — measure before committing.

Framework-Specific Notes

pipeline

Pipelines accept the same truncation controls — pass them through:

from transformers import pipeline
 
clf = pipeline("sentiment-analysis")
clf(long_text, truncation=True, max_length=512)   # no warning, no crash

sentence-transformers

SentenceTransformer truncates automatically at model.max_seq_length, so you usually won't crash — but you will silently lose the tail of long inputs. Set the limit deliberately:

from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer("all-MiniLM-L6-v2")
print(model.max_seq_length)     # e.g. 256
model.max_seq_length = 384      # raise up to the model's real ceiling, not beyond

For documents longer than the ceiling, chunk first and average the chunk embeddings — the model will not do that for you.

LangChain

A frequent cause here is measuring chunk size in characters while the model counts tokens. Use a token-aware splitter so chunks actually fit:

from langchain.text_splitter import RecursiveCharacterTextSplitter
 
splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(
    tokenizer, chunk_size=500, chunk_overlap=50,
)

For debugging related LangChain issues, see resolving LangChain ValidationError.

Verification Steps

1. Print the token length after tokenizing: enc["input_ids"].shape[-1] should be <= your max_length.
2. Run one over-long sample end to end — the previously-crashing input should now pass through the model cleanly.
3. Confirm no warning prints. If it still does, you are tokenizing somewhere without truncation/max_length — search your code for every tokenizer( and .encode( call.
4. On GPU, rerun on CPU once. CUDA asserts are vague; the CPU IndexError names the real problem. If CPU is clean, GPU will be too.
5. Spot-check truncated content so you didn't cut off something important — if you did, switch to chunking or a long-context model.

Prevention

• Always pass truncation=True and an explicit max_length.
• Read the true limit from model.config.max_position_embeddings.
• Guard against the model_max_length sentinel on custom tokenizers.
• Use token-aware splitters, never character counts, for RAG chunking.
• Choose truncation vs chunking deliberately per task — don't lose the tail by accident.
• Add an assertion in preprocessing: assert ids.shape[-1] <= max_len.
• Only silence the warning after every path is provably length-safe.

Troubleshooting Matrix

Related Guides

• Resolve CUDA error: device-side assert triggered
• Resolve OSError: Can't load tokenizer
• Fix OpenAI RateLimitError (429)
• Resolve LangChain ValidationError
• Chunking strategies for RAG pipelines

External References

• Hugging Face — Padding and truncation
• Hugging Face — Tokenizer API
• Longformer model docs
• PyTorch — nn.Embedding

FAQs

Does the warning mean my data was already truncated? No — the opposite. The warning means the tokenizer did not truncate; it kept every token. Silent truncation is a different situation (for example sentence-transformers cutting at max_seq_length), and it does not print this warning.

Why do I only see this warning sometimes? Hugging Face throttles it — it typically prints once per tokenizer instance, on the first over-long sequence. Later over-long sequences in the same run may not re-print it, which is why it is easy to miss during batch preprocessing.

Is 512 always the right max_length? Only for models trained with 512 positions (BERT, RoBERTa). GPT-2 is 1024, T5 is 512 by default, Longformer is 4096. Always take the number from model.config.max_position_embeddings, not a habit.

Can I just catch the IndexError and continue? Don't. On GPU a device-side assert corrupts the CUDA context, so subsequent operations fail unpredictably even after you catch it. Prevent the over-long sequence from reaching the model instead of handling the crash.

What's the difference between max_length and model_max_length? model_max_length is the tokenizer's stored idea of the model's ceiling (and may be an unset sentinel). max_length is the per-call limit you pass to tokenizer(...). When you pass max_length, it wins — which is exactly why passing it explicitly is the reliable fix.