Fix RuntimeError: CUDA out of memory (PyTorch)

The familiar failure:

RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0;
15.78 GiB total capacity; 13.52 GiB already allocated; 1.23 GiB free;
13.90 GiB reserved in total by PyTorch)

The message is not a generic instruction to call torch.cuda.empty_cache(). It is an allocation report. The useful fix depends on whether another process owns the GPU, the workload's live tensors genuinely exceed capacity, a graph is being retained, or the allocator cannot reuse its cached blocks effectively.

This guide starts with measurement, then applies the smallest durable change. It covers training, validation, LLM workloads, Docker, cloud GPUs, Windows, Linux, WSL, and distributed jobs.

Quick Answer

PyTorch CUDA out of memory means a requested GPU allocation cannot fit because another process is using VRAM, live tensors reach capacity, retained graphs leak memory, or cached blocks are fragmented. Inspect nvidia-smi, allocated, reserved, and peak memory first. Then reduce batch or sequence length, use AMP, gradient accumulation or checkpointing, and tune the allocator only when evidence supports it.

TL;DR

• Start with nvidia-smi. A second notebook, worker, or scheduled job may own the memory you thought was free.
• memory_allocated() measures live PyTorch tensors; memory_reserved() also includes reusable cache. A high reserved number alone is not a leak.
• If peak allocated memory nearly fills the card, reduce the micro-batch, sequence length, image resolution, or model footprint.
• Use gradient accumulation to preserve effective batch size. Use AMP and activation checkpointing to reduce activation memory with different speed tradeoffs.
• model.eval() is not enough for evaluation. Use torch.inference_mode() or torch.no_grad() and detach values kept for metrics or logging.
• Use PYTORCH_ALLOC_CONF before the process begins when data points to native allocator fragmentation. PYTORCH_CUDA_ALLOC_CONF is its legacy alias.
• empty_cache() cannot free live tensors or reduce a workload's true peak. It is not a per-batch remedy.

Symptoms

Common Causes

Root Cause: Peak Memory, Not Average Memory

The decisive number is the highest simultaneous allocation. A model can fit between steps and still fail during forward, backward, optimizer step, or a single unusually long batch.

For training, peak GPU memory is roughly made of:

For transformers, activation memory rises with batch size and tokens. Standard attention can also make the sequence-length term much more expensive than linear. That is why reducing a 16,384-token context can be a stronger OOM fix than reducing a model's hidden size slightly. For vision models, image resolution and feature-map size play the same role.

Allocated, reserved, and what nvidia-smi sees

PyTorch uses a caching allocator. When a tensor is freed, PyTorch often keeps the underlying block instead of immediately returning it to the CUDA driver, making later allocations faster.

• allocated is memory holding live tensors in the current PyTorch process.
• reserved is memory the PyTorch allocator has obtained, including inactive blocks it may reuse.
• nvidia-smi reports device-level usage across processes and driver-managed allocations. It will not mirror one PyTorch counter exactly.

If allocated is near capacity, your workload needs less live memory. If reserved is far above allocated, do not conclude “leak” from that gap alone: the cache may be healthy. It is a signal to inspect fragmentation, variable shapes, and allocator backend only after checking the workload and other processes.

Step-by-Step Solution

Step 1: Capture evidence before changing code

Run this at a known point in the job, such as immediately after a training step. It measures the current process and the selected CUDA device:

import torch
 
 
def report_cuda_memory(device: int = 0) -> None:
    torch.cuda.synchronize(device)
    allocated = torch.cuda.memory_allocated(device) / 1024**3
    reserved = torch.cuda.memory_reserved(device) / 1024**3
    peak = torch.cuda.max_memory_allocated(device) / 1024**3
    free, total = torch.cuda.mem_get_info(device)
 
    print(f"allocated: {allocated:.2f} GiB")
    print(f"reserved:  {reserved:.2f} GiB")
    print(f"peak:      {peak:.2f} GiB")
    print(f"driver free: {free / 1024**3:.2f} GiB")
    print(f"driver total: {total / 1024**3:.2f} GiB")
 
 
torch.cuda.reset_peak_memory_stats(0)
# Run one representative training or inference step here.
report_cuda_memory(0)

The peak value is more informative than a single idle reading. Reset it between phases to discover whether the forward pass, backward pass, validation, or generation step is the peak. Run nvidia-smi in a second terminal too:

nvidia-smi

Read the PID, command, and GPU memory columns. Stop a stale process you own only after confirming its command and owner. On shared servers, do not kill a different user's job; request an unoccupied device through the scheduler.

Step 2: Rule out device selection and external pressure

On Linux, nvidia-smi is usually on the PATH. On Windows, run nvidia-smi.exe from PowerShell. In WSL, the Windows NVIDIA driver must expose the GPU to the Linux distribution; nvidia-smi should show the same physical device and active processes. macOS does not support NVIDIA CUDA locally, so a CUDA OOM there normally belongs to a remote Linux machine or container.

In Docker, containers are not a VRAM isolation boundary by default. Two containers using the same physical device compete for the same frame-buffer memory. Assign devices deliberately and inspect from the host when possible:

docker run --gpus '"device=0"' --rm my-training-image python train.py

In Kubernetes, a GPU request should allocate a device to the pod, but MIG can assign a memory-limited slice rather than the full card. Record the visible GPU name and total memory at job start. Cloud notebook runtimes may also retain old kernels after a browser tab closes.

Step 3: Reduce the highest-memory tensor dimension

Do not lower learning rate or randomly call garbage collection. Neither changes the tensor shapes that caused the allocation. Reduce one material dimension, measure again, and keep the smallest change that leaves operating headroom.

For variable-length text, the batch's maximum padded length matters. Bucketing examples of similar lengths, capping input tokens, and rejecting pathological records provide a more stable peak than simply choosing a smaller global batch.

Step 4: Preserve effective batch size with gradient accumulation

Gradient accumulation cuts activation memory because backward runs on a small micro-batch. The optimizer updates after several micro-batches, approximating a larger effective batch.

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
 
 
device = torch.device("cuda")
features = torch.randn(256, 128)
labels = torch.randint(0, 10, (256,))
dataset = TensorDataset(features, labels)
 
micro_batch_size = 4
accumulation_steps = 8
loader = DataLoader(
    dataset,
    batch_size=micro_batch_size,
    shuffle=True,
    drop_last=True,
)
 
model = nn.Sequential(nn.Linear(128, 256), nn.ReLU(), nn.Linear(256, 10)).to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)
 
model.train()
optimizer.zero_grad(set_to_none=True)
for step, (inputs, targets) in enumerate(loader):
    logits = model(inputs.to(device, non_blocking=True))
    loss = nn.functional.cross_entropy(logits, targets.to(device, non_blocking=True))
    (loss / accumulation_steps).backward()
 
    if (step + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)

This example uses drop_last=True so every optimizer update has the same number of micro-batches. If your last group is partial, explicitly rescale it or flush it with the intended gradient normalization. Accumulation preserves the effective batch of 4 × 8 = 32 but not every training dynamic: batch-norm statistics, scheduler timing, and gradient clipping may need adjustment.

Step 5: Enable modern automatic mixed precision

AMP often reduces activation memory and can improve throughput on supported NVIDIA GPUs. Current PyTorch APIs live under torch.amp; older torch.cuda.amp examples remain common online.

import torch
 
 
def train_one_epoch(model, loader, optimizer) -> None:
    scaler = torch.amp.GradScaler("cuda")
    model.train()
 
    for inputs, targets in loader:
        inputs = inputs.cuda(non_blocking=True)
        targets = targets.cuda(non_blocking=True)
        optimizer.zero_grad(set_to_none=True)
 
        with torch.autocast(device_type="cuda", dtype=torch.float16):
            logits = model(inputs)
            loss = torch.nn.functional.cross_entropy(logits, targets)
 
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

For hardware and models that support it, bfloat16 is often more numerically forgiving than float16 and generally does not need gradient scaling. Replace the autocast dtype deliberately and validate loss stability, accuracy, and overflow behavior. AMP lowers many activation costs; it does not magically compress every parameter, optimizer state, temporary workspace, or KV cache.

Step 6: Checkpoint activations when compute is available

Activation checkpointing stores fewer intermediate results during forward and recomputes selected regions in backward. It is a strong option for deep transformers and vision backbones when the GPU has compute headroom.

import torch
from torch import nn
from torch.utils.checkpoint import checkpoint
 
 
class CheckpointedNetwork(nn.Module):
    def __init__(self, block: nn.Module, classifier: nn.Module) -> None:
        super().__init__()
        self.block = block
        self.classifier = classifier
 
    def forward(self, inputs: torch.Tensor) -> torch.Tensor:
        hidden = checkpoint(self.block, inputs, use_reentrant=False)
        return self.classifier(hidden)

Use checkpointing around expensive blocks, not every trivial operation. It trades memory for additional forward computation during backward and can increase wall-clock time. The PyTorch checkpoint documentation explains behavior, RNG preservation, and reentrant versus non-reentrant implementations.

Step 7: Fix retained graphs and evaluation leaks

model.eval() changes layers such as dropout and batch normalization. It does not turn off gradients. Use inference mode for a true evaluation-only path:

import torch
 
 
@torch.inference_mode()
def validate(model: torch.nn.Module, loader: torch.utils.data.DataLoader) -> float:
    was_training = model.training
    model.eval()
    correct = 0
    total = 0
 
    for inputs, targets in loader:
        logits = model(inputs.cuda(non_blocking=True))
        predictions = logits.argmax(dim=1)
        correct += (predictions.cpu() == targets).sum().item()
        total += targets.numel()
 
    model.train(was_training)
    return correct / total

When logging training values, keep Python numbers or CPU tensors, not the autograd graph:

# Incorrect: each loss retains its backward history.
loss_history.append(loss)
 
# Correct: keep a Python scalar.
loss_history.append(loss.item())
 
# Correct for outputs needed later: remove graph history and GPU ownership.
saved_logits.append(logits.detach().cpu())

If memory climbs a little every step, find the reference before adding del or gc.collect(). Those calls only help after the real owner has released the tensor. Callback state, metric objects, closures, progress bars, and lists of predictions are common owners.

Step 8: Tune the allocator only when the evidence fits

The PyTorch caching allocator improves speed by keeping blocks. Allocator tuning is most useful when a variable-shape workload shows a large reserved-versus- allocated gap or reports inactive split blocks, not when allocated memory itself fills the card.

For current releases, set the configuration before importing or initializing CUDA:

PYTORCH_ALLOC_CONF=expandable_segments:True python train.py

PYTORCH_CUDA_ALLOC_CONF remains a backwards-compatible alias. On Windows PowerShell:

$env:PYTORCH_ALLOC_CONF = "expandable_segments:True"
python train.py

The options are not interchangeable:

expandable_segments and max_split_size_mb are native-allocator controls and do not solve all backends or all types of fragmentation. Read the current PyTorch CUDA memory-management documentation before deploying allocator settings fleet-wide.

Step 9: Scale a model that cannot fit one GPU

When parameters, gradients, optimizer state, and a minimal micro-batch still do not fit, batch-size tuning has reached its limit.

Distributed Data Parallel replicates the model on each rank; it does not make one rank's model copy smaller. FSDP and ZeRO use sharding to address that footprint. See PyTorch FSDP before introducing sharding: checkpointing, optimizer setup, and rank-local data handling all change.

empty_cache(): What It Does and Does Not Do

torch.cuda.empty_cache() releases inactive cached blocks from the current process back to the CUDA driver. It can lower memory_reserved(), make memory available to another process, and occasionally help a narrow fragmentation case after the real tensor owners have been released.

It does not:

• free live tensors tracked by memory_allocated();
• lower the peak required by the same model, batch, and input shapes;
• fix a graph retained in a Python container; or
• make per-iteration cache clearing a performance-neutral practice.

This is a reasonable boundary between two unrelated workloads in one process after releasing objects. It is not a training-loop ritual. Measure before and after instead of assuming it repaired the source of an OOM.

Verification Steps

1. Run nvidia-smi before the job and confirm the selected GPU, owner, and total memory are expected.
2. Reset peak stats, execute one representative worst-case step, and record allocated, reserved, peak, and driver free memory.
3. Run a full train-and-validation epoch. Validation must use inference mode or no-grad and memory must not creep upward across batches.
4. Test the longest expected text, largest image, or highest concurrency case, not only a friendly sample.
5. If you change AMP, checkpointing, quantization, or sharding, verify loss, accuracy, checkpoint restore, and throughput as well as memory.
6. If you tune allocator settings, compare the exact same seed and input-shape distribution with and without the setting.
7. Keep a meaningful safety margin. A job that uses the final few megabytes is fragile when a library workspace, a longer input, or another process appears.

Prevention

Memory budget checklist

• Log max_memory_allocated() for representative training, validation, and inference paths.
• Bound text tokens, image dimensions, and concurrent generation requests.
• Bucket variable-length data so a short record does not share a batch with one extreme record.
• Make AMP, evaluation inference mode, and scalar metric logging default code-review checks.
• Establish one job owner per assigned GPU or a scheduler-enforced sharing policy.
• Record PyTorch version, CUDA runtime, driver, GPU model, and allocator config with every benchmark.
• Profile a production-shaped batch before deploying a new model or context length.

Platform notes

Troubleshooting Matrix

Related Guides

• fix bitsandbytes CUDA setup failures
• resolve CUDA device-side asserts
• fix torch.cuda.is_available() returning false
• fix token sequences that exceed a model limit
• fix CUDA library loading errors in TensorFlow

Official References

• PyTorch CUDA memory management
• PyTorch CUDA environment variables
• PyTorch automatic mixed precision
• PyTorch activation checkpointing
• PyTorch Fully Sharded Data Parallel
• NVIDIA nvidia-smi documentation

FAQs

What causes RuntimeError: CUDA out of memory in PyTorch?

The GPU could not satisfy a tensor allocation. Usual causes are a micro-batch, sequence, image, model, optimizer state, or KV cache that requires too much live memory; another process consuming VRAM; a graph or tensor retained across steps; or, less often, allocator fragmentation. The allocated, reserved, peak, and device-level process numbers identify which one applies.

Does torch.cuda.empty_cache() fix CUDA out of memory?

Not when live tensors create the peak. It releases unused cached blocks, so it can reduce reserved memory for another process and may help a narrow fragmentation case. It cannot free live allocations, fix retained graphs, or make an oversized workload fit. Do not place it in every iteration.

Why does PyTorch run out of CUDA memory during validation?

Validation may retain autograd graphs if it runs without torch.inference_mode() or torch.no_grad(). model.eval() alone does not disable gradients. Also avoid retaining GPU logits and losses in metric lists; use item() for scalars or detach().cpu() for tensors needed after the loop.

What is the difference between allocated and reserved GPU memory in PyTorch?

Allocated memory holds live tensors. Reserved memory is memory held by PyTorch's caching allocator, including reusable inactive blocks. A gap is not automatically a leak. If allocated nearly reaches capacity, reduce live memory. If reserved is far higher than allocated, investigate fragmentation and variable-shape workloads before changing allocator settings.

How do I keep batch size while reducing CUDA memory?

Lower the micro-batch and use gradient accumulation: divide each micro-batch loss by the accumulation count, backpropagate each one, and update the optimizer after the group. This retains a similar effective batch while the peak activation memory follows the smaller micro-batch.

Which allocator setting fixes PyTorch CUDA fragmentation?

Set PYTORCH_ALLOC_CONF before Python starts. expandable_segments:True can help changing allocation sizes with the native allocator, while max_split_size_mb is a measured last resort for inactive split blocks. PYTORCH_CUDA_ALLOC_CONF is an alias. None of these settings creates VRAM or fixes allocated memory that already fills the GPU.

Sources

• PyTorch — CUDA semantics & memory management