Fix "bitsandbytes CUDA Setup failed"

You try to load a model in 8-bit or 4-bit (QLoRA, load_in_4bit=True) and bitsandbytes stops you:

RuntimeError: CUDA Setup failed despite GPU being available. Please run the
following command to get more information: python -m bitsandbytes

Older versions dump a wall of CUDA SETUP: lines and, crucially, a line like:

CUDA SETUP: Loading binary .../libbitsandbytes_cpu.so...

That _cpu.so line is the whole story. bitsandbytes ships different compiled binaries for different CUDA versions. When it can't find one that matches your setup, it falls back to the CPU-only binary — which can't run GPU quantization, so it raises this error.

So this is not really a mysterious CUDA bug. It's "bitsandbytes couldn't find its GPU binary." This guide fixes that in the order that resolves the most cases with the least effort — and the first real fix is usually just an upgrade.

Quick Answer

bitsandbytes CUDA Setup failed means no matching CUDA binary was found, so bitsandbytes loaded the CPU library. Fix it in order: confirm torch.cuda.is_available() is True (bitsandbytes needs a CUDA PyTorch), pip install -U bitsandbytes (resolves most modern cases), then if libcudart still isn't found, add the CUDA libs to LD_LIBRARY_PATH or set BNB_CUDA_VERSION.

TL;DR

• The signal: libbitsandbytes_cpu.so loaded → no CUDA binary matched.
• Fix 1 (most cases): pip install -U bitsandbytes.
• Prerequisite: a CUDA build of PyTorch — torch.cuda.is_available() must be True.
• libcudart missing? Add CUDA libs to LD_LIBRARY_PATH, run ldconfig.
• Version mismatch / multiple toolkits? BNB_CUDA_VERSION=122 + matching path.
• Windows: upgrade to official wheels; the old bitsandbytes-windows fork is dead.
• Diagnose: python -m bitsandbytes prints a full setup report.

The Signal: Which Binary Loaded

When you import bitsandbytes, it detects your GPU and the CUDA version that PyTorch was built with, then looks for a compiled binary named like libbitsandbytes_cuda121.so. Two outcomes:

• Match found → the CUDA binary loads, and 8-bit/4-bit kernels work.
• No match → it loads libbitsandbytes_cpu.so and reports CUDA Setup failed.

So the fix is always "make a matching CUDA binary loadable." Everything below is a different reason that match is failing.

First, Run the Built-in Diagnostic

bitsandbytes ships a self-check. Run it before changing anything:

python -m bitsandbytes

It reports your PyTorch CUDA version, the detected GPU, which binary it's trying to load, and whether it found libcudart. Read the last few lines — they usually name the exact problem (no CUDA torch, missing libcudart, or version mismatch).

Step-by-step Solution

Step 1: Confirm PyTorch actually has CUDA

bitsandbytes rides on PyTorch's CUDA. If PyTorch itself is CPU-only, bitsandbytes can never succeed — fix that first:

import torch
print(torch.cuda.is_available())   # must be True
print(torch.version.cuda)          # must be a version, not None

If this is False or None, stop here and fix PyTorch — see fixing torch.cuda.is_available() returns False. Chasing bitsandbytes while PyTorch is CPU-only wastes hours.

Step 2: Upgrade bitsandbytes (the fix for most cases)

Older bitsandbytes had fragile CUDA detection and shipped binaries for only a handful of CUDA versions. Modern releases detect the environment far better and bundle binaries for the full CUDA 11–12 range, so an upgrade alone resolves the majority of these reports:

pip install -U bitsandbytes
python -m bitsandbytes    # re-check: should now find a CUDA binary

• -U → upgrade to the latest release.
• If you're on a locked version for another dependency, upgrade to the newest one that dependency allows — even a minor bump often fixes detection.

Step 3: Make libcudart findable

If the diagnostic says libcudart.so not found in any environmental path, bitsandbytes can't locate the CUDA runtime. Point the loader at it:

# Find the CUDA runtime library
find / -name "libcudart.so*" 2>/dev/null
 
# Add its directory to the library path (adjust the path to your CUDA install)
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
sudo ldconfig            # refresh the linker cache

• LD_LIBRARY_PATH → tells the dynamic loader where to find libcudart.
• ldconfig → rebuilds the linker cache so the change takes effect.
• Put the export in your shell profile or job script so it persists.

Step 4: Override the CUDA version for mismatches

If you have multiple CUDA toolkits installed, or bitsandbytes picks a version that doesn't match what's actually available, override the detection:

export BNB_CUDA_VERSION=122     # e.g. force the CUDA 12.2 binary
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-12.2/lib64

• BNB_CUDA_VERSION=122 → forces bitsandbytes to load the cuda122 binary instead of PyTorch's default.
• Always pair it with an LD_LIBRARY_PATH that actually contains that version's libraries, or you've just picked a binary that still can't find its runtime.

Platform-Specific Notes

Windows

The historical answer — "install the bitsandbytes-windows fork" — is obsolete and wrong now. Current bitsandbytes provides official Windows wheels:

pip uninstall -y bitsandbytes bitsandbytes-windows
pip install -U bitsandbytes

The LD_LIBRARY_PATH and ldconfig steps are Linux-only. On Windows, a current wheel plus a working CUDA PyTorch is normally all you need.

WSL2

Treat WSL2 as Linux, but the driver comes from Windows (see the WSL notes in the torch.cuda guide). Once nvidia-smi and torch.cuda.is_available() work inside WSL2, upgrade bitsandbytes and it typically resolves.

Docker / clusters

• Base your image on an official CUDA image (nvidia/cuda:...-runtime or PyTorch's CUDA image) so libcudart is present.
• Run with --gpus all.
• On HPC clusters, module load cuda/12.x before running, and confirm you're on a GPU node — a login node has no GPU and bitsandbytes will fall back to CPU.

Colab / Kaggle

Usually just an upgrade: pip install -U bitsandbytes, then restart the runtime so the new binary is imported. Re-running the cell without restarting keeps the old, failed import cached.

Verification Steps

1. python -m bitsandbytes reports a CUDA binary loaded (not _cpu.so) and no errors.
2. A real 4-bit load succeeds:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
 
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)
model = AutoModelForCausalLM.from_pretrained(
    "facebook/opt-350m", quantization_config=bnb, device_map="auto",
)
print(next(model.parameters()).device)   # cuda:0

3. The model's parameters report cuda:0, and nvidia-smi shows GPU memory in use.
4. No CUDA Setup failed and no _cpu.so line on import.

The Diagnostic Ladder

Follow it top to bottom — each rung is more effort than the last, and most people never get past rung 2:

1. torch.cuda.is_available() False? Fix PyTorch first.
2. bitsandbytes out of date? pip install -U bitsandbytes.
3. libcudart not found? LD_LIBRARY_PATH + ldconfig.
4. Multiple toolkits / mismatch? BNB_CUDA_VERSION + matching path.
5. Still failing? Reinstall in the active environment.

Prevention

• Pin a recent bitsandbytes in requirements; don't inherit an ancient one.
• Install a CUDA PyTorch from the correct index before bitsandbytes.
• In Docker, start from an official CUDA base image so libcudart exists.
• Keep bitsandbytes, PyTorch, and the driver upgrades together, not piecemeal.
• Run python -m bitsandbytes in CI to catch a broken setup before deploy.
• On Windows, use official wheels — never the deprecated fork.
• Restart the runtime/kernel after upgrading so the new binary is imported.

Troubleshooting Matrix

Related Guides

• Fix torch.cuda.is_available() returns False
• Resolve CUDA error: device-side assert triggered
• Resolve OSError: Can't load tokenizer
• Resolve Hugging Face 401 Unauthorized
• A practical guide to 4-bit QLoRA fine-tuning

External References

• bitsandbytes documentation
• bitsandbytes — installation & troubleshooting
• bitsandbytes releases (GitHub)
• PyTorch — Get Started (install selector)

FAQs

Why does the error say "despite GPU being available"? Because bitsandbytes can see a GPU — the driver and nvidia-smi are fine — but it still couldn't load a CUDA binary that matches. The GPU being present rules out "no hardware"; the failure is in the binary/runtime match, not the card.

I upgraded and it still fails — what next? Run python -m bitsandbytes and read the last lines. If it names libcudart, do Step 3. If it shows a version mismatch, do Step 4. Also confirm you upgraded in the same environment you're running — a different venv is a common miss.

Can I run bitsandbytes without a GPU? Not for GPU quantization. The CPU fallback exists but does not provide the 8-bit and 4-bit CUDA kernels, so load_in_4bit/load_in_8bit require a working CUDA setup.

Does bitsandbytes need to match my PyTorch CUDA version exactly? It needs a compatible binary, not a byte-exact match. Modern releases bundle a range and detect the right one. When detection is wrong, BNB_CUDA_VERSION lets you force the correct binary.

Is python -m bitsandbytes safe to run in production images? Yes — it's a read-only diagnostic. Running it during the Docker build is a good way to fail fast if the CUDA setup is broken before the image ships.