aifromzero.dev

Make It Fit a Free GPU: QLoRA from Scratch (Fine-Tuning in Production, Module 5)

Module 5 of 15 14 min read Lab code ↗

This is Module 5 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Start at Module 1 or browse the full syllabus.

You switched to LoRA and it still says CUDA out of memory

In Module 4 you did everything right. You wrapped Anvil’s base model in LoRA adapters, froze the original weights, and dropped the trainable parameter count from 4 billion to roughly 1%. The adapter is a few megabytes. On paper, training just got cheap.

Then you open a free Colab notebook, pick the T4 runtime, run the same script, and watch it die on the first step:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB.
GPU 0 has a total capacity of 14.74 GiB of which 38.06 MiB is free.

Here’s the trap nobody mentions when they sell you on LoRA: LoRA shrinks what you train, not what you load. The 4 billion parameters of Qwen3-4B-Base are still sitting in VRAM in bfloat16 — about 8 GB before a single activation exists — because the forward and backward passes still need to run through the whole network. The adapter rides on top of that 8 GB; it doesn’t replace it. On a 16 GB T4, with activations, gradients, and optimizer state stacked on, you run out of room.

The fix is not “rent an A100.” The fix is to quantize the frozen base model to 4-bit while you keep training the adapters in full precision. That’s QLoRA, and it’s what turns this course from “you’ll need a decent card” into “this runs on a free Colab T4.” By the end of this module Anvil trains comfortably under 16 GB, you’ll have measured the actual peak VRAM yourself, and you’ll know exactly where every gigabyte goes — and, just as important, why a QLoRA-trained model is not something you can deploy as-is.

In this module

You’ll learn how to:

  1. Explain why LoRA on a 4B base still won’t comfortably fit a free 16 GB T4 — and what exactly eats the VRAM: base weights, activations, gradients, and optimizer states.
  2. Build a QLoRA setup from scratch — load Qwen3-4B-Base in 4-bit NF4 with BitsAndBytesConfig, call prepare_model_for_kbit_training, then attach the same LoRA adapters from Module 4.
  3. Reason about the QLoRA recipe internals — NF4 vs int8, double quantization, the compute dtype, and paged optimizers — and where each one buys you memory (paper 2305.14314).
  4. Budget and debug GPU memory: read a VRAM breakdown, and apply the OOM remedies (gradient checkpointing, smaller batch, gradient accumulation, paged optimizers) in the right order.
  5. Distinguish training-time 4-bit quantization (QLoRA) from inference quantization (GGUF/AWQ/GPTQ, Module 12) — and explain why a QLoRA-trained model is not a deployment artifact.

You’ll build: Anvil now trains on a free Colab T4. config.py gains tier="qlora" — Qwen3-4B-Base loaded in 4-bit NF4 with double quantization and a bfloat16 compute dtype, prepared for k-bit training, with your Module-4 LoRA adapters on top. You’ll measure the actual VRAM, hit (and fix) an OOM, and see your lab.md print the real time and cost of the run.

Theory areas covered:

  • T5 — QLoRA & training-time quantization — 10% of the course. Module 5 owns this area; it’s reinforced later by Unsloth (M6), inference quantization (M12), and the capstone (M15).

This module stands on T3 (SFT) and T4 (LoRA), both of which you’ve already built. We reuse the SFTTrainer/SFTConfig from Module 3 unchanged and the LoraConfig from Module 4 unchanged. The only thing that changes is how the base model is loaded.

Prerequisites: Modules 1–4 — the installed stack, config.py’s get_model_and_tokenizer, the SFTTrainer/SFTConfig from Module 3, the LoraConfig/get_peft_model from Module 4, and the tool-calling dataset from Module 2. For the lab you need a free Colab or Kaggle GPU (T4, 16 GB) — that’s the default path — or any local NVIDIA GPU with ≥ 16 GB. You’ll need HF_TOKEN configured (Module 1). One hard constraint: bitsandbytes 4-bit kernels are CUDA-only — there’s no CPU or Apple Silicon path for the real run. (The offline smoke test sidesteps this; more on that in the lab.)

Where Anvil stands

From here on, every module marks where you are with ✅ (done), 👉 (you’re here), and ⬜ (ahead):

  • ✅ M1 — When (Not) to Fine-Tune. Baseline: the base model fails the tool-call format.
  • ✅ M2 — Data Is the Model. data.py: the tool-calling dataset, formatted via the chat template.
  • ✅ M3 — Your First Fine-Tune: SFT. train_sft.py: Anvil v0, SFTTrainer/SFTConfig.
  • ✅ M4 — LoRA: Train 1% of the Weights. LoraConfig, swappable adapters.
  • 👉 M5 — Make It Fit a Free GPU: QLoRA. 4-bit NF4 base + LoRA → Anvil fits a free T4.
  • ⬜ M6 — Go Faster: Unsloth and Config-as-Code (Axolotl). The same QLoRA, ~2× faster.
  • ⬜ M7 — Is It Any Better? Task-Grounded Evaluation.
  • ⬜ M8–M9 — Align It: DPO and the preference family.
  • ⬜ M10 — Teach It to Reason: GRPO and Verifiable Rewards.
  • ⬜ M11 — Model Merging with mergekit.
  • ⬜ M12 — Shrink It for Serving: GGUF, AWQ, GPTQ.
  • ⬜ M13 — Serve It: vLLM, Ollama, TGI.
  • ⬜ M14 — Ship It: Publishing and Model Cards.
  • ⬜ M15 — Capstone: Anvil, End to End.

Anvil’s state before this module: it trains in LoRA, but the full bfloat16 base barely fits a T4 — the 8 GB of base weights is the whole problem. After this module: it trains in 4-bit QLoRA, comfortably on a free Colab T4, with the peak VRAM measured and the cost printed. The tier="qlora" you freeze here is the foundation of the free path — reused (and accelerated) by Unsloth in Module 6, reused again for GRPO in Module 10, and replayed in the capstone.

Why LoRA alone doesn’t fit: the VRAM budget

Start where Module 4 left off. LoRA freezes the base model and trains two small low-rank matrices — B·A — per adapted layer. Only about 1% of the parameters update. That’s a real, dramatic reduction. But it’s a reduction in trainable parameters, and trainable is not the same as resident. The full base model still has to live in GPU memory because every forward pass runs activations through all 4 billion parameters, and the backward pass propagates gradients back through them to reach the adapters that sit on top.

So where does the VRAM actually go during a fine-tune? There are four consumers, and it’s worth knowing each one cold, because the OOM remedies later in this module each attack a specific one:

  1. Base model weights. 4B parameters × 2 bytes (bfloat16) ≈ 8 GB, just to hold the network. This is fixed the instant you load the model, before you process a single token.
  2. Activations. The intermediate tensors saved during the forward pass so the backward pass can compute gradients. These scale with batch size × sequence length — the one consumer you can dial with hyperparameters.
  3. Gradients. One gradient per trainable parameter. In full fine-tuning that’s 4B gradients (another ~8–16 GB). In LoRA it’s gradients for the ~1% adapter params only — tiny.
  4. Optimizer states. AdamW keeps two moments (a running mean and variance) per trained parameter. In full FT that’s two more copies of the model — enormous. In LoRA, again, only for the ~1% adapter params — tiny.

Here’s the key insight, and it’s the whole reason QLoRA exists. LoRA already crushed consumers (3) and (4): gradients and optimizer state, which dominate full fine-tuning, are negligible once you’re only training adapters. What LoRA does nothing about is consumer (1) — the 8 GB of frozen base weights. And on a 16 GB T4, that 8 GB plus activations is exactly what tips you over. “Switching to LoRA” doesn’t fit a 4B model on 16 GB because LoRA wasn’t aimed at the consumer that’s overflowing.

flowchart TB
    subgraph budget["16 GB T4 budget"]
      direction LR
      subgraph lora["tier = full + LoRA (bf16 base)"]
        direction TB
        l1["base weights<br/>~8 GB (bf16)"]
        l2["activations<br/>~4-6 GB"]
        l3["LoRA grads + AdamW<br/>~0.5 GB"]
        l4["⚠️ total ≈ 13-15 GB<br/>(OOM on a spike)"]
      end
      subgraph qlora["tier = qlora (4-bit base)"]
        direction TB
        q1["base weights<br/>~2.5 GB (4-bit NF4)"]
        q2["activations<br/>~4-6 GB"]
        q3["LoRA grads + AdamW<br/>~0.5 GB"]
        q4["✅ total ≈ 7-9 GB<br/>(comfortable headroom)"]
      end
    end
    note["LoRA shrinks what you TRAIN.<br/>QLoRA shrinks what you LOAD."]

The cure is to attack consumer (1) directly: QLoRA = a frozen base model quantized to 4-bit, plus LoRA adapters trained in full precision. Quantizing the base from 16-bit to 4-bit cuts that 8 GB block to roughly 2.5 GB. Everything else stays the same. That’s the difference between OOM and comfortable headroom.

This isn’t a hack someone improvised — it’s the central result of the QLoRA paper (“QLoRA: Efficient Finetuning of Quantized LLMs,” Dettmers et al., arXiv 2305.14314, NeurIPS 2023). The paper showed you could fine-tune a 65B model on a single 48 GB GPU — a setup that previously needed a multi-GPU rig — with no measurable loss in quality versus 16-bit LoRA. That headline result is what makes a free 16 GB T4 enough for a 4B model: if 65B fits in 48 GB, a 4B model fits in 16 GB with room to spare.

The QLoRA recipe: NF4, double quantization, and the compute dtype

QLoRA is three quantization ideas stacked on top of LoRA. In code they’re four arguments to one object. Anvil keeps that object in exactly one place — a helper in config.py — because, per the fil rouge contract, every quantization and PEFT decision lives in config.py and nowhere else. Here’s the canonical config, verbatim from the lab’s anvil/config.py:

def qlora_bnb_config():
    """The bitsandbytes 4-bit config that makes Anvil fit a free 16 GB T4 (Module 5, QLoRA)."""
    import torch
    from transformers import BitsAndBytesConfig

    return BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=torch.bfloat16,
    )

Four knobs. Each one is doing something specific, and each is worth understanding at the “why,” not just the “what.”

load_in_4bit=True tells bitsandbytes to store the base model’s weights in 4 bits instead of 16. This happens on the fly at load time — there’s no calibration step, no dataset, no separate quantization pass. You hand from_pretrained the config and the weights come down quantized. That on-the-fly, no-calibration property is exactly what separates training-time quantization from the inference quantization you’ll meet in Module 12, and it’s the heart of this module’s big distinction. (As of bitsandbytes 0.49.2, June 2026, this config object is in transformers itself — from transformers import BitsAndBytesConfig.)

bnb_4bit_quant_type="nf4" picks the type of 4-bit number. NF4 (4-bit NormalFloat) is the paper’s signature contribution: a 4-bit data type that is information-theoretically optimal for weights that follow a normal distribution — which, conveniently, neural-network weights do. The intuition: a naive 4-bit type spaces its 16 representable values evenly, but if your values cluster around zero in a bell curve, you’re wasting precision out in the tails where almost nothing lives. NF4 places its 16 levels so that each one is equally likely to be used given a normal distribution, so you spend bits where the weights actually are. The alternative 4-bit type, fp4, exists but is generally worse for this; int8 is a different thing entirely (more on that in the table below).

bnb_4bit_use_double_quant=True turns on double quantization — quantizing the quantization constants themselves. Here’s what that means. To dequantize a 4-bit weight back to a usable number you need a scaling constant per block of weights. Those constants are stored in higher precision and, across millions of blocks, they add up to real memory. Double quantization quantizes them too, saving on average about 0.4 bits per parameter (the paper’s figure). That sounds trivial, and on a big GPU it is — but on a 16 GB T4 where you’re counting hundreds of megabytes, a few hundred MB of headroom is the difference between a batch size of 1 and a batch size of 2. Turn it on.

bnb_4bit_compute_dtype=torch.bfloat16 is the one people misread, so read it twice. The weights are stored in 4-bit, but the actual matrix multiplications do not happen in 4-bit. For every matmul, bitsandbytes dequantizes the relevant 4-bit weights back up to bfloat16 on the fly, does the math in bfloat16, and discards the dequantized copy. So 4-bit is a storage format here, not a compute format. The compute dtype is what precision the math runs in once the weights are unpacked. bfloat16 is the right choice on modern GPUs.

⚠️ A note on the T4 and bfloat16. The Tesla T4 is a Turing-architecture card, and Turing predates native bfloat16. You’ll sometimes see T4 recipes use torch.float16 instead. In practice, as of bitsandbytes 0.49.2 (June 2026), the 4-bit kernels run fine with bnb_4bit_compute_dtype=torch.bfloat16 on a T4 — the dequantization path handles it — so the lab keeps bfloat16 for consistency with the rest of the course. If you ever see precision warnings or instability on older hardware, switching the compute dtype to float16 is the first thing to try. Don’t take either as gospel; check it against the wheel you actually installed.

With the config built, you load the model and prepare it. This is the exact sequence in get_model_and_tokenizer(tier="qlora") — and the order matters:

from transformers import AutoModelForCausalLM
from peft import prepare_model_for_kbit_training

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, dtype=torch.bfloat16, quantization_config=qlora_bnb_config())
model = prepare_model_for_kbit_training(model)   # NOT prepare_model_for_int8_training

prepare_model_for_kbit_training is a small but load-bearing step. It takes the freshly quantized, frozen base model and gets it ready to receive gradients on the adapters you’re about to attach. Concretely it casts the layer norms and the output head to float32 for numerical stability, enables gradient checkpointing on the model, makes the input embeddings require gradients (so the backward pass can flow through to the adapters), and disables the KV cache (which is for inference, not training). Think of it as “make this k-bit frozen model trainable-on-top.”

⚠️ Freshness trap. It is prepare_model_for_kbit_training, never the old prepare_model_for_int8_training. The int8 function was removed from PEFT back in v0.10, but it lives on in thousands of stale notebooks and blog posts. As of peft 0.19.1 (June 2026), prepare_model_for_kbit_training is the one and only correct call. If you copy a tutorial that uses the int8 name, it will simply ImportError.

Now the LoRA adapters go on top — and this is the part you do not re-learn, because it’s Module 4 unchanged. The same LoraConfig, the same get_peft_model:

from peft import LoraConfig, get_peft_model

lora = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                  lora_dropout=0.05, bias="none", task_type="CAUSAL_LM")
model = get_peft_model(model, lora)

That’s the whole trick in one sentence: a 4-bit frozen base plus bf16-trained adapters is QLoRA. The LoraConfig is identical to the one you wrote in Module 4. The only thing that changed between LoRA and QLoRA is how the base model was loaded — full-precision bfloat16 before, 4-bit NF4 now. The adapter math doesn’t know or care.

flowchart LR
    x([input tokens]) --> base
    subgraph fwd["forward + backward pass"]
      base["Qwen3-4B-Base weights<br/>FROZEN · stored 4-bit NF4"]
      base -->|"dequantize on the fly → bf16"| mm["matmul (bf16 compute)"]
      lora["LoRA adapters B·A<br/>TRAINABLE · bf16"] --> mm
      mm --> out([logits])
    end
    out -. "∇ loss" .-> lora
    note["4-bit is a STORAGE format here — not a compute format,<br/>and not a deployment format (see Module 12)."]

Trace the gradient in that diagram: the loss flows back, but it only updates the LoRA adapters. The 4-bit base weights are frozen — they’re dequantized to do the forward math, but they never change. That’s why the optimizer state stays tiny (consumer 4 from the budget) and why a 4-bit base costs you almost no accuracy: the high-precision adapters absorb all the task-specific learning.

NF4 isn’t int8 — and int8 isn’t QLoRA

The single most common QLoRA mistake on the internet is reaching for load_in_8bit=True and calling the result “QLoRA.” It is not. int8 quantization is a real, useful thing that bitsandbytes supports — but it’s a different precision, a different memory profile, and crucially, it is not the QLoRA recipe. QLoRA is specifically 4-bit NF4. Here’s the contrast:

load_in_8bit=True (int8)load_in_4bit=True + nf4 (QLoRA)
Bits per parameter84
4B base footprint~4 GB~2.5 GB (with double quant)
Data type rationaleLinear int8; fine, but coarserNF4: optimal for normally-distributed weights
Calibration required?No — on the flyNo — on the fly
Is this QLoRA?NoYes

int8 halves the base footprint; 4-bit NF4 quarters it (a bit more, with double quant) and is the configuration the QLoRA paper validated. On a 16 GB T4 that extra ~1.5 GB of headroom is not optional — it’s what lets you run a real batch. So when you see a tutorial set load_in_8bit=True under a “QLoRA” heading, you’ve found a stale tutorial. (As of bitsandbytes 0.49.2, June 2026, both load_in_8bit and load_in_4bit exist; only the 4-bit NF4 path is QLoRA.)

Fitting 16 GB: the VRAM budget in practice and OOM remedies

The spirit of this course is honest numbers, not hand-waving, so the lab measures VRAM rather than asserting it. The pattern is two lines around a training step:

import torch
torch.cuda.reset_peak_memory_stats()
# ... run a training step (or the whole trainer.train()) ...
peak_gb = torch.cuda.max_memory_allocated() / 1e9
print(f"peak VRAM: {peak_gb:.2f} GB")

reset_peak_memory_stats() zeroes the high-water mark; max_memory_allocated() reads it back after the work is done. On my run of the lab — Qwen3-4B-Base in 4-bit NF4, a small batch, sequence length capped at the config’s max_length=1024 — the peak landed comfortably under 16 GB, where the same run as train_sft(tier="full", use_lora=True) (the Module-4 LoRA path, full bfloat16 base) had been bumping the ceiling. Treat that as “on my run,” not a constant: your exact peak depends on batch size, sequence length, the wheel you installed, and what else is on the card. The point is the shape of the number, and that you measured it yourself.

What actually fits on a free 16 GB T4? Here’s the teachable table:

SetupBase weights footprintFits on a 16 GB T4?Note
Full fine-tune, 4B~8 GB weights + ~16 GB grads/optimizer❌ NoOptimizer state alone blows the budget
LoRA, 4B (bf16 base)~8 GB weights⚠️ TightFits in theory; OOMs on an activation spike
QLoRA, 4B (4-bit NF4)~2.5 GB weights✅ YesThe happy path — comfortable headroom
QLoRA, 7–8B (4-bit NF4)~4–5 GB weights✅ YesStill fits — the reason this course is “free to learn”

That last row is the fact that makes the whole course possible: a QLoRA fine-tune of a 1–8B model fits inside 16 GB. That’s why the default path of this course is a free Colab or Kaggle T4, not a rented GPU.

Now, what do you do when, despite QLoRA, your run still spikes into an OOM? There’s a correct order to the remedies, and the order matters because the cheap, low-cost fixes come first and the ones that change training dynamics come with caveats. Here’s how I work down the list on a T4:

1. Gradient checkpointing — your first lever. Recall that activations (consumer 2) scale with batch and sequence length and can dominate. Gradient checkpointing trades memory for time: instead of keeping every layer’s activations resident for the backward pass, it keeps only a few checkpoints and recomputes the rest during the backward pass. You give up roughly 20–30% in speed and get a large activation-memory reduction in return. It’s enabled with gradient_checkpointing=True on the SFTConfig (and prepare_model_for_kbit_training already nudges it on for the model). This is lever number one because it’s the biggest single win.

2. Reduce per_device_train_batch_size. Fewer sequences in flight means fewer resident activations. This is the most direct lever — halve the batch, roughly halve the activation memory. The catch is in lever three.

3. Gradient accumulation — recover the effective batch. Dropping the batch from 8 to 1 doesn’t just save memory; it changes the training dynamics, because the optimizer now updates on a batch of 1 instead of 8. Gradient accumulation fixes that for free: you accumulate gradients over several micro-batches before taking one optimizer step, so the effective batch size = per_device_train_batch_size × gradient_accumulation_steps. A per_device_train_batch_size=1 with gradient_accumulation_steps=8 gives you an effective batch of 8 at the memory cost of 1. (This reinforces the SFT hyperparameters from Module 3 — same fields, same SFTConfig.) The distinction to hammer: shrinking the batch without accumulating changes how the model learns; accumulating restores it.

4. Paged optimizers — the spike catcher. The last remedy is the paper’s own contribution: paged optimizers. Set optim="paged_adamw_8bit" (or "paged_adamw_32bit") on the SFTConfig. bitsandbytes then pages the optimizer states out to CPU RAM when a memory spike threatens to OOM the GPU, much like an operating system pages memory to disk. This won’t lower your steady-state usage, but it stops a transient spike — say, on an unusually long sequence — from killing a run that was otherwise fitting. It’s the safety net, not the foundation, which is why it’s last.

My default on a T4: start at per_device_train_batch_size=1, gradient_accumulation_steps=8, gradient checkpointing on. That’s an effective batch of 8 and it leaves headroom. I only add optim="paged_adamw_8bit" if I still see spikes. In TRL these are all fields on the SFTConfig — for example, config.sft_args(gradient_checkpointing=True, per_device_train_batch_size=1, gradient_accumulation_steps=8) — never arguments on the SFTTrainer itself. (As of trl 1.6.0, June 2026, SFTConfig carries gradient_checkpointing, optim, and gradient_accumulation_steps; passing them to the Trainer would TypeError.)

And to be clear about the escalation path: the sequence is LoRA-on-bf16 OOMs on a T4 → switch to tier="qlora" → if it still spikes, drop the batch, accumulate, and page the optimizer. It is not “rent an A100.” The rented-GPU path (which we’ll cost out in a moment) is for genuinely heavy jobs like GRPO in Module 10 — not for a 4B QLoRA SFT, which is built to run free.

QLoRA is not a deployment format: training-quant vs inference-quant

This is the most important conceptual distinction in the entire course, and it’s the one that trips up almost everyone. You just trained Anvil with load_in_4bit=True. It is tempting — and wrong — to conclude that you now have “a 4-bit model” ready to ship. You do not. To see why, you have to separate two completely different uses of the word “quantization.”

Training-time quantization (QLoRA — what you just did). bitsandbytes quantizes the frozen base to 4-bit NF4, on the fly at load time, with no calibration. During training those 4-bit weights are dequantized back to bfloat16 for every matmul. The goal is narrow: make the training run fit in VRAM. What comes out the other end is a frozen base plus a set of separate LoRA adapter weights — there is no single “4-bit model file” on disk anywhere.

Inference-time quantization (PTQ — GGUF, AWQ, GPTQ, Module 12). Here you take an already-trained model and quantize it for serving — to make it small and fast on a target runtime. This is post-training quantization (PTQ), often with a calibration step (a small dataset used to pick quantization parameters that minimize error — k-quant GGUF being a notable exception that needs none), producing a real file format that a serving runtime like llama.cpp, Ollama, or vLLM loads directly.

Training-time (QLoRA)Inference-time (PTQ)
Library / formatbitsandbytes 4-bit NF4GGUF / AWQ / GPTQ
WhenDuring the fine-tuneAfter training, to serve
Calibration?No — on the flyOften yes
PurposeMake training fit in VRAMServe small / fast
Artifact4-bit base in memory + adapterA servable model file
ModuleM5 (here)M12

So why is a QLoRA-trained Anvil not deployable as-is? Because the 4-bit weights only ever existed in memory, quantized by bitsandbytes at load time for training. There’s no exported 4-bit artifact, and your task-specific knowledge lives in separate adapter weights sitting on top of a frozen base. To actually serve Anvil you’ll do two distinct things in Module 12: first merge_and_unload — fold the adapter back into a full-precision base so you have one self-contained model — and then quantize that for inference (to GGUF, AWQ, or GPTQ). That second quantization has a different purpose, a different method, and a different output than the QLoRA one. They share a word and almost nothing else.

⚠️ Common misconception: “I trained with load_in_4bit, so my model is already a 4-bit model I can deploy.”

No. The 4-bit in QLoRA is a training-time storage trick — the weights are dequantized back to bfloat16 to do the actual math, and nothing 4-bit is ever written to disk. It is neither a servable file format nor inference-time PTQ. To deploy, you first merge_and_unload the adapter into a full-precision base, then quantize for inference (GGUF/AWQ/GPTQ — Module 12). Corollary, same family of error: load_in_8bit is not “QLoRA” either. QLoRA is 4-bit NF4, full stop.

Build it: Anvil trains on a free T4

Goal: make Anvil’s fine-tune fit a free 16 GB Colab/Kaggle T4 by flipping config.py to tier="qlora" — Qwen3-4B-Base in 4-bit NF4 plus LoRA — measure the peak VRAM, trigger and fix an OOM, and record the real time and cost.

What you’ll see: the run’s loss goes down, and the final line prints the measured peak VRAM (peak VRAM: ~X.XX GB) comfortably under 16 GB — where train_sft(tier="full", use_lora=True) (the Module-4 LoRA path on a full bfloat16 base) had been OOMing or grazing the ceiling.

The full, tested code lives in finetune-prod-labs/module-05. The offline smoke test runs with no GPU and no network, so you can verify the plumbing before you spend a single GPU-minute.

Step 1 — Open a T4 runtime and install the GPU extra. On Colab or Kaggle, pick a T4 GPU runtime. The repo locks the core training stack (torch, transformers, trl, peft, datasets, accelerate) in its shared pyproject.toml; the CUDA-only piece for this module — bitsandbytes — is installed per-runtime:

pip install bitsandbytes==0.49.2   # CUDA only — the QLoRA 4-bit kernels

Set HF_TOKEN in the environment (a .env file or a Colab secret), then confirm you actually got a T4:

import torch
print(torch.cuda.get_device_name())   # -> "Tesla T4"

Step 2 — Flip to tier="qlora". This is the one real change, and it’s already wired in anvil/config.py. The qlora_bnb_config() helper (shown earlier) builds the canonical 4-bit NF4 config, and get_model_and_tokenizer("qlora") loads the base with it and calls prepare_model_for_kbit_training:

elif tier == "qlora":
    model_id, local = MODEL_ID, False
    kwargs = {"dtype": torch.bfloat16, "quantization_config": qlora_bnb_config()}
    kbit = True
# ... later, if kbit:
from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)

The BitsAndBytesConfig lives only here — no quantization config anywhere else in the codebase. That’s the frozen contract: one place for every quantization and PEFT decision.

Step 3 — Train with the Module-3 script, unchanged. You don’t write a new training loop. train_sft.py already takes a tier, and QLoRA is just tier="qlora" with LoRA on:

from anvil.train_sft import train_sft

trainer = train_sft(tier="qlora", use_lora=True,
                    gradient_checkpointing=True, per_device_train_batch_size=1,
                    gradient_accumulation_steps=8)

Those three overrides flow straight onto the SFTConfig (TRL v1 idioms: max_length and the training fields live on the SFTConfig, and the tokenizer is passed as processing_class to the Trainer — never tokenizer= or max_seq_length). The trainer wires the LoRA peft_config automatically because use_lora=True.

Step 4 — Measure the VRAM. Wrap the run with the peak-memory probe from earlier — reset_peak_memory_stats() before, max_memory_allocated() after — and print the number. Compare it (in your notes) against train_sft(tier="full", use_lora=True) — the Module-4 LoRA path on the full bfloat16 base, which on a T4 sits right against the ceiling.

Step 5 — Trigger and fix an OOM. Deliberately break it: crank per_device_train_batch_size up and turn gradient_checkpointing off, and watch torch.OutOfMemoryError: CUDA out of memory appear. Then fix it in the order from this module — checkpointing on, batch back to 1, accumulation to 8, and if it still spikes, optim="paged_adamw_8bit". Each lever is one line on the SFTConfig.

Step 6 — Record the cost. Note the wall-clock time of the run and write it into the frozen cost line at the bottom of lab.md. On the free path the dollar figure is $0.00 — that’s the whole point of the pivot module.

Step 7 — Run the offline tests. From the repo root:

uv run pytest module-05/tests/

The smoke test runs on CPU with no network. It can’t exercise the real 4-bit load — bitsandbytes needs CUDA — so it does two things instead: it asserts the bitsandbytes config is built correctly (NF4 + double quant + bf16 compute), and it re-confirms that LoRA still trains end-to-end on the tiny fixture (because QLoRA is that LoRA run, on a 4-bit base). The real 4-bit load is a @pytest.mark.gpu test, skipped automatically when no CUDA device is present.

def test_bnb_config_is_nf4_double_quant():
    bnb = config.qlora_bnb_config()
    assert bnb.load_in_4bit is True
    assert bnb.bnb_4bit_quant_type == "nf4"
    assert bnb.bnb_4bit_use_double_quant is True
    assert bnb.bnb_4bit_compute_dtype == torch.bfloat16


@pytest.mark.gpu
def test_qlora_loads_4bit_on_gpu():
    model, tok = config.get_model_and_tokenizer("qlora")
    assert tok.pad_token is not None

What this lab does NOT do, on purpose. No Unsloth, no Axolotl — that’s Module 6; here we stay on bare TRL + PEFT + bitsandbytes, the canonical path. No merge_and_unload, no GGUF/AWQ/GPTQ — that’s Module 12; we pose the train-vs-inference distinction but export no deployment format. No DPO/GRPO (Module 8+), no formal base-vs-tuned eval (Module 7) — we just watch the loss go down. And no load_in_8bit masquerading as QLoRA — it appears once, in the table above, as a contrast, never as the recipe.

Try it yourself:

  1. Swap MODEL_ID for a larger 7–8B Qwen3 base (if one is available to you) and confirm the QLoRA fine-tune still fits the T4 (the 1–8B happy path). Compare the peak VRAM to the 4B run.
  2. Measure the time overhead of QLoRA versus LoRA in bfloat16 (on a local GPU with ≥ 16 GB, or a short rented A100 run). That gap is the price of dequantizing 4-bit → bf16 on every matmul — the “free VRAM, paid in time” trade.

In production

QLoRA is your free on-ramp, but free in VRAM is not free in time. Dequantizing 4-bit weights back to bfloat16 on every matmul adds compute, so a QLoRA run is slower than the same run on a full-precision base that already fits. That’s the explicit trade: you spend wall-clock to buy VRAM. The moment that trade stops being worth it, the cost calculation becomes explicit.

A QLoRA fine-tune of a 7B model will fit a free T4 — but slowly, in the hours. The alternative is a rented GPU: roughly 2–4 hours on a rented A100 ≈ $1.50–$3 (as of June 2026, RunPod ~$1.19/h, Lambda ~$1.99/h, Modal ~$2.50/h billed by the second — these prices move weekly; check each provider’s page before you spend). The decision threshold is simple: if iteration is slow enough to break your flow, the rented GPU pays for itself fast. I keep a free-T4 run to smoke-test the config and rent an A100 only for the final, longer run — the $2 buys me hours back.

A few production habits worth keeping. Pin bitsandbytes exactly (bitsandbytes==0.49.2, not a floating >=): the on-disk meaning of a quantized model can shift between versions, and you don’t want a silent change under a long run. Measure the peak VRAM before you commit to a long job — the two-line probe costs nothing and saves you from discovering an OOM three hours in. And keep gradient checkpointing plus paged optimizers as a standing safety net on memory-tight hardware.

One forward look: Module 6 runs this same QLoRA through Unsloth, which claims roughly 2× faster training and up to 70% less VRAM (its own figure, as of unsloth 2026.6.7, June 2026 — take it as reported, not promised). It’s an optional accelerator. The canonical path — the one you understand end to end because you built it — stays this bare TRL/PEFT/bitsandbytes recipe.

Concept check

Why this matters. This module gives you the lever that makes the rest of the course free: QLoRA. The skills are (1) knowing why LoRA alone overflows a T4 — it shrinks what you train, not what you load; (2) reading the recipe — what NF4, double quantization, and the compute dtype each buy you; (3) applying the OOM remedies in the right order; and (4) the conceptual keystone — train-time quantization is not inference quantization, and a QLoRA model is not a deployable artifact. The tier="qlora" you froze here is the foundation reused all the way to the capstone. The questions below stay inside what this module taught.

  1. A teammate says “I’m doing QLoRA — I set load_in_8bit=True.” What’s wrong, and what should they write instead?

    • A. Nothing’s wrong; load_in_8bit is the standard QLoRA setting.
    • B. int8 is not QLoRA; QLoRA is load_in_4bit=True, bnb_4bit_quant_type="nf4". int8 exists but isn’t the QLoRA recipe.
    • C. They should use load_in_16bit=True for QLoRA.
    • D. QLoRA doesn’t use BitsAndBytesConfig at all.

  2. A LoRA fine-tune of a 4B model in bfloat16 OOMs on a 16 GB T4, even though only ~1% of the parameters are trainable. Why, and what’s the fix?

    • A. LoRA has a memory leak; restart the runtime.
    • B. The ~1% adapter gradients are the problem; lower the LoRA rank.
    • C. The full base weights (~8 GB in bf16) dominate VRAM — LoRA shrinks what you train, not what you load. Load the base in 4-bit (QLoRA).
    • D. The optimizer states for the full model are the problem; switch optimizers.

  3. You want to serve your QLoRA-trained Anvil. You point a serving runtime at the training output and it won’t load as a deployable model. What do you need to do first, and why?

    • A. Nothing — a QLoRA model is already a 4-bit deployable file.
    • B. merge_and_unload the adapter into a full-precision base, then quantize for inference (GGUF/AWQ/GPTQ). The QLoRA 4-bit is a training-time storage trick (dequantized to bf16 to compute); no servable file exists.
    • C. Re-run training with load_in_8bit=True to produce a deployable model.
    • D. Just upload the adapter; serving runtimes load raw adapters as 4-bit models.

  4. Your QLoRA run still spikes into OOM on a T4. In what order do you apply the remedies, and which one trades VRAM for time?

    • A. Paged optimizers first, then everything else; none trades VRAM for time.
    • B. Gradient checkpointing (trades VRAM for ~25% time) → reduce batch size → gradient accumulation to recover the effective batch → paged optimizers for spikes.
    • C. Reduce the LoRA rank first, then increase batch size.
    • D. Switch to float16 weights, then raise max_length.

  5. A developer believes that because the base is “4-bit,” the matrix multiplications run in 4-bit. Correct them — what is bnb_4bit_compute_dtype for?

    • A. They’re right; 4-bit storage means 4-bit compute.
    • B. 4-bit is a storage format; weights are dequantized to bnb_4bit_compute_dtype (e.g. bfloat16) for every matmul, and the math runs at that precision.
    • C. bnb_4bit_compute_dtype sets the on-disk file format of the saved model.
    • D. It controls the LoRA adapter’s rank.

Answers.

  1. B. QLoRA is specifically 4-bit NF4 (load_in_4bit=True, bnb_4bit_quant_type="nf4"), plus double quantization and a bf16 compute dtype. int8 (load_in_8bit) is a real bitsandbytes feature but a different precision and not the QLoRA recipe — calling it QLoRA is the single most common mistake in stale tutorials.
  2. C. Trainable ≠ resident. LoRA crushes gradients and optimizer state (the things full FT struggles with), but it does nothing about the ~8 GB of bfloat16 base weights that must stay loaded for the forward/backward pass. That block is what overflows a 16 GB T4. Loading the base in 4-bit NF4 — QLoRA — cuts it to ~2.5 GB and you fit.
  3. B. The 4-bit in QLoRA only ever existed in memory during training, dequantized to bf16 to compute; nothing 4-bit was written to disk, and your task knowledge lives in separate adapter weights. To deploy you first merge_and_unload the adapter into a full-precision base, then run a separate inference quantization (GGUF/AWQ/GPTQ) — that’s Module 12, a different quantization with a different purpose.
  4. B. Gradient checkpointing first (biggest single win; recomputes activations in the backward pass, costing ~20–30% time) — that’s the one that trades VRAM for time. Then drop the batch size, then add gradient accumulation to restore the effective batch you lost (effective batch = per_device_train_batch_size × gradient_accumulation_steps), and finally paged optimizers to catch transient spikes.
  5. B. 4-bit is purely a storage format. For every matmul, bitsandbytes dequantizes the weights up to the compute dtype — bfloat16 here — and the math runs at that precision; the dequantized copy is then discarded. The compute dtype is the precision of the actual arithmetic, not the storage and not the on-disk format.

Common pitfalls.

  • Calling load_in_8bit “QLoRA.” QLoRA is load_in_4bit + nf4 + double quantization. int8 is a different (coarser, larger) thing. A “QLoRA” tutorial that sets load_in_8bit=True is stale. (As of bitsandbytes 0.49.2, June 2026.)
  • Reaching for prepare_model_for_int8_training. It was removed from PEFT in v0.10 and will ImportError. The current call is prepare_model_for_kbit_training. It’s all over old notebooks — verify against peft 0.19.1, not against a 2023 blog post.
  • “My QLoRA model is already a deployable 4-bit model.” The conceptual trap of the module. The 4-bit is a training-time storage trick (compute happens in bf16); deploying means merge_and_unload then inference quantization (Module 12). Train-quant and inference-quant share a word and nothing else.

Key takeaways

  • LoRA shrinks what you train; QLoRA shrinks what you load. On a 4B model the ~8 GB of bfloat16 base weights is what overflows a 16 GB T4 — and LoRA doesn’t touch it.
  • QLoRA = a frozen 4-bit NF4 base + LoRA adapters trained in bf16. Only the base loading changes from Module 4; the LoraConfig and the SFTTrainer run are identical.
  • The canonical config has four knobs: load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16 — and the order is from_pretrained(quantization_config=...) → prepare_model_for_kbit_training → get_peft_model.
  • It’s prepare_model_for_kbit_training, never prepare_model_for_int8_training (removed in PEFT v0.10).
  • OOM remedies in order: gradient checkpointing → smaller batch → gradient accumulation (to restore the effective batch) → paged optimizers (for spikes). Gradient checkpointing trades VRAM for time.
  • A QLoRA fine-tune of a 1–8B model fits in 16 GB — that’s the free path this whole course rides on.
  • A QLoRA model is not a deployment artifact. 4-bit is a training-time storage format (dequantized to bf16 to compute). To serve: merge_and_unload, then inference quantization (GGUF/AWQ/GPTQ, Module 12) — train-quant ≠ inference-quant.

What’s next

Anvil fits a free GPU now — but the run is slow. In Module 6 you’ll run this same QLoRA recipe through Unsloth to get it roughly 2× faster, and you’ll see the whole pipeline expressed as a single Axolotl YAML config. Both are accelerators you can take or leave; the bare TRL/PEFT/bitsandbytes path you built here stays the one you fully understand.

Want the full fine-tuning concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · ← Module 4: LoRA: Train 1% of the Weights · Module 6: Go Faster: Unsloth and Config-as-Code → · Lab code for this module

References