Resolve "CUDA error: device-side assert triggered"

You are training on GPU, everything runs for a while, then:

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so
the stacktrace below might be incorrect. For debugging consider passing
CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side
assertions.

Two things make this error miserable. First, the traceback points at some innocent line — often loss.item() or .to(device) — that has nothing to do with the real bug. Second, once it fires, everything on that GPU keeps failing, so people restart, hit it again, and conclude the GPU is broken.

Neither is true. This is almost always a plain out-of-range index — a label or a token id that is too big — and CUDA's asynchronous execution is just hiding where. This guide shows you how to force the real line to surface, then how to fix the handful of things that actually cause it.

Quick Answer

CUDA error: device-side assert triggered means a GPU kernel hit a failed assertion — nearly always an out-of-range index. Because CUDA runs asynchronously, the traceback is misleading. Restart the process, then rerun on CPU or with CUDA_LAUNCH_BLOCKING=1 to get the true line, and check that your CrossEntropyLoss targets are in [0, num_classes-1] or your embedding ids are below the vocabulary size.

TL;DR

• It's an out-of-range index, not a hardware or driver fault.
• The traceback lies because CUDA kernels run asynchronously.
• Get the real line: rerun on CPU, or set CUDA_LAUNCH_BLOCKING=1.
• Top cause: CrossEntropyLoss target outside [0, num_classes-1].
• Second cause: embedding id >= vocab size, or sequence longer than the model's positions.
• Restart the process. The CUDA context is corrupted — you cannot catch and continue.

Why the Traceback Lies

CUDA kernels are launched asynchronously. When your Python code calls a GPU operation, it queues the work and returns immediately — it does not wait for the GPU to finish. So if a kernel launched at your loss computation faults, Python has already moved on. The error only becomes visible at the next point that synchronises with the GPU — commonly loss.item(), .cpu(), print(tensor), or the next epoch's first CUDA call.

That is why the traceback blames the wrong line. It is reporting where the GPU caught up, not where the fault happened.

There are two ways to fix the reporting:

Get the Real Error

Option 1: CUDA_LAUNCH_BLOCKING=1

Set it before the process starts — it must be read before CUDA initialises:

CUDA_LAUNCH_BLOCKING=1 python train.py

Or at the very top of your script, before any torch CUDA call:

import os
os.environ["CUDA_LAUNCH_BLOCKING"] = "1"   # must run before CUDA initialises
 
import torch
# ... rest of your code

Now the traceback points at the true line — usually your loss or an embedding call.

Option 2: Rerun on CPU

The CPU has no asynchronous queue, so it raises a precise Python error at the exact operation:

import torch
 
device = "cpu"   # temporarily force CPU
model = model.to(device)
# inputs/targets also to CPU
 
out = model(inputs)
loss = criterion(out, targets)
# e.g. IndexError: Target 10 is out of bounds.  ← the real message

On CPU, CrossEntropyLoss will say Target 10 is out of bounds, and an embedding will say index out of range in self. Those messages name the problem directly.

Root Cause

The GPU asserted because a kernel tried to read memory outside a tensor's bounds. Three operations cause the overwhelming majority of cases:

Step-by-step Solution

Cause 1: Label out of range (the #1 case)

nn.CrossEntropyLoss expects class indices, not one-hot vectors, and they must run from 0 to num_classes - 1. The classic breakers:

• Labels start at 1 (so the max label equals num_classes).
• A dataset has more classes than the model's final layer outputs.
• A sentinel like -1 sits in the labels where ignore_index should be -100.

Validate before the loss so you get a clean assertion on the right line:

num_classes = model.classifier.out_features   # or however many logits you output
 
assert targets.min() >= 0, f"negative label: {targets.min().item()}"
assert targets.max() < num_classes, (
    f"label {targets.max().item()} >= num_classes {num_classes}"
)
 
loss = criterion(logits, targets)

• targets.min() >= 0 → catches negative labels and bad sentinels.
• targets.max() < num_classes → catches off-by-one and class-count mismatches.
• These run on whatever device the tensors are on and fail with a readable message.

Cause 2: Embedding index out of range

nn.Embedding(num_embeddings=V, ...) has exactly V rows. Any input id >= V indexes past the table. This is the same failure as an over-long token sequence, one stage later — see the token-length warning guide.

vocab_size = model.get_input_embeddings().num_embeddings
 
assert input_ids.max() < vocab_size, (
    f"token id {input_ids.max().item()} >= vocab {vocab_size}"
)
assert input_ids.min() >= 0, "negative token id"

Common triggers:

• A tokenizer and model from different checkpoints (mismatched vocab sizes).
• You added tokens with tokenizer.add_tokens(...) but forgot model.resize_token_embeddings(len(tokenizer)).
• Sequences longer than max_position_embeddings (position ids overflow).

If you added tokens, resize the model's embeddings to match:

tokenizer.add_tokens(["<new_token>"])
model.resize_token_embeddings(len(tokenizer))   # keep model and tokenizer in sync

Cause 3: Wrong loss for the output shape

Using the wrong loss also asserts or produces invalid indices:

• Binary problem with CrossEntropyLoss: feed raw logits and integer targets 0/1; do not sigmoid first.
• Multi-label problem: use BCEWithLogitsLoss with float targets in [0,1], not CrossEntropyLoss.
• NLLLoss: expects log_softmax inputs; pair it correctly or use CrossEntropyLoss, which does the log_softmax for you.

Verification Steps

1. Restart the process so you have a clean CUDA context before testing the fix.
2. Reproduce on CPU once. The previously-crashing batch should now pass with no IndexError.
3. Keep the assertions in during development — they turn a future silent bug into a clear message on the right line.
4. Run one full epoch on GPU without CUDA_LAUNCH_BLOCKING. If it completes, the async error is gone.
5. Log label and id ranges for the first few batches: targets.min()/max() and input_ids.max() should sit inside the valid bounds.

The Debug Ladder

Follow it in order every time and the error stops being mysterious:

1. Restart — the context is corrupted, so nothing else is reliable until you do.
2. Get the real line — CPU rerun or CUDA_LAUNCH_BLOCKING=1.
3. Identify the op — loss/criterion vs embedding/indexing, from the true traceback.
4. Validate the range — assert labels in [0, C-1] or ids < vocab size.

Platform-Specific Notes

Jupyter / Kaggle / Colab

Re-running a cell does not reset the CUDA context. After a device-side assert you must restart the kernel (Kernel → Restart), or every subsequent GPU cell will keep failing with the same error even after you fix the bug.

Multi-GPU / DataParallel

The assertion message may name a specific block and thread on one device. The bug is still an out-of-range index; isolate it by running a single-GPU, CPU, or CUDA_LAUNCH_BLOCKING=1 pass to get a clean trace before touching the parallel setup.

TORCH_USE_CUDA_DSA

The error text suggests compiling with TORCH_USE_CUDA_DSA. That enables device-side assertions in a source build of PyTorch for richer kernel-level diagnostics. For everyday debugging you almost never need it — CUDA_LAUNCH_BLOCKING=1 plus a CPU rerun is faster and enough.

Troubleshooting Matrix

Related Guides

• Fix "Token indices sequence length is longer than the specified maximum"
• Fix torch.cuda.is_available() returns False
• Resolve OSError: Can't load tokenizer
• Fix OpenAI RateLimitError (429)
• Debugging PyTorch training loops

External References

• PyTorch — CUDA semantics (asynchronous execution)
• PyTorch — CrossEntropyLoss
• PyTorch — nn.Embedding
• Hugging Face — resize_token_embeddings

FAQs

Is this error a sign my GPU is broken? Almost never. It is a software-level assertion — an out-of-range index in your data or model config. A genuine hardware fault usually shows as Xid errors in dmesg or ECC failures, not a device-side assert.

Why does it sometimes appear only after many steps? The out-of-range value only exists in some batches. Training runs fine until the first batch containing a bad label or an over-long sequence arrives — then it asserts. Validate ranges for every batch, not just the first.

Can I wrap the training step in try/except and skip bad batches? No. After the assert the CUDA context is corrupted, so the except block runs on a dead context and the next CUDA call fails anyway. Prevent the bad index before it reaches the GPU instead.

Does CUDA_LAUNCH_BLOCKING=1 fix the error? No — it only makes the traceback accurate. It changes when the error is reported, not whether it happens. Use it to find the bug, then fix the underlying index.

My labels look fine but it still asserts — what now? Rerun on CPU for the exact message. If it names an embedding, check input_ids against vocab size and sequence length against max_position_embeddings. If it names the loss, print targets.unique() — a single stray value is usually hiding in there.