aifromzero.dev

Align It: Preference Tuning with DPO (Fine-Tuning in Production, Module 8)

Module 8 of 15 15 min read Lab code ↗

This is Module 8 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Start at Module 1 or browse the full syllabus.

Your model knows what good looks like. It doesn’t know what better looks like.

In Module 7 you proved it. Anvil — your Qwen3-4B base model, supervise-fine-tuned on tool-calling demonstrations — beats the raw base model on every metric that matters: it emits valid JSON tool calls, it picks the right function name, it fills in the right arguments. The base model couldn’t do any of that. The numbers are in the table; SFT works.

And yet. On some prompts Anvil still hesitates. Two tools are plausible — a search_notes and a search_emails — and it picks the less relevant one. Or it gets the call right but wraps it in a paragraph of friendly preamble — “Sure! I’d be happy to help with that. Let me look that up for you…” — when the only thing your runtime wants is the call. The behavior isn’t wrong, exactly. It’s just not the one you’d have chosen.

Here’s why, and it’s the whole module in one sentence: SFT only ever showed Anvil good answers. It never showed it a bad one. Supervised fine-tuning minimizes a next-token loss on demonstrations — it teaches the model to reproduce targets. It has no notion of “better than.” It can’t learn to prefer the concise call over the chatty one, or the relevant tool over the plausible-but-wrong one, because it was never shown the two side by side and told which one wins.

To teach that, you need to show the model both — a chosen answer and a rejected one for the same prompt — and optimize it to prefer the chosen. That’s preference tuning, and in 2026 the default method is not the heavy RLHF-PPO pipeline of 2022. It’s DPO — Direct Preference Optimization — a single, stable, supervised-style loss with no reward model and no online reinforcement-learning loop. This module aligns Anvil to prefer the right tool call, on a free Colab T4, and then re-runs Module 7’s eval to prove the alignment actually helped.

In this module

You’ll learn how to:

  1. Explain what SFT cannot teach — imitation has no notion of “better than” — and why preference tuning is the step that shapes behavior after SFT (it doesn’t add knowledge, and it doesn’t replace SFT).
  2. Contrast RLHF-PPO (the InstructGPT pipeline: SFT → reward model → online PPO loop) with DPO, and explain why DPO became the 2026 default: no separate reward model, no online RL rollouts, one stable loss, cheaper and more reproducible.
  3. Build preference data — turn your tool-calling fixture into (prompt, chosen, rejected) triples, a valid concise call as chosen versus a degraded one as rejected — and know where pairs come from in real projects.
  4. Train Anvil with the stable DPOTrainer + DPOConfig: start from the SFT checkpoint, let TRL manage the frozen reference model, tune beta, and read the DPO loss and reward margins — TRL v1 idioms only (processing_class, never tokenizer=).
  5. Re-evaluate with Module 7’s eval.py (DPO-vs-SFT on the held-out, plus an anti-forgetting glance) and diagnose the classic DPO failure modes (too-high beta, a drifting reference, reward hacking toward verbosity).

You’ll build: Anvil now learns to prefer the right tool call, not just imitate one. You’ll build (prompt, chosen, rejected) triples from the fixture, add anvil/train_dpo.py (the stable DPOTrainer with DPOConfig, starting from your SFT checkpoint with a frozen reference model and a tuned beta), then re-run Module 7’s eval.py to show the DPO-vs-SFT delta — all on a free Colab T4.

Theory areas covered:

  • T8 — Preference tuning (DPO) — 11% of the course. T8 is shared with Module 9: this module owns the DPO half (the method, its founding paper, the reference model, beta); Module 9 owns the family — ORPO, KTO, SimPO — and the decision tree.

This module also leans hard on two areas you’ve already built: T3 (SFT) — DPO starts from your SFT checkpoint, and the reference model is that checkpoint, frozen — and T7 (evaluation) — we re-run eval.py unchanged to measure the gain, never assume it.

Prerequisites: Modules 1–7. Specifically: the pinned stack from Module 1, config.get_model_and_tokenizer(tier="qlora") from Module 5, the SFT checkpoint from Module 3, the tool-calling dataset from Module 2, and — most of all — eval.py from Module 7, which we re-execute. For the lab: a free Colab/Kaggle T4 (16 GB) is the default path (a QLoRA DPO run on a 4B model fits comfortably); a local GPU with ≥ 16 GB works too. Have HF_TOKEN set (Module 1).

Where you are

From now on, every module marks where you are with ✅ (done), 👉 (you’re here), and ⬜ (ahead):

  • ✅ M1–M7 — baseline, data, SFT, LoRA, QLoRA, Unsloth/Axolotl, and task-grounded evaluation. Anvil is a working SFT model with a proven base-vs-fine-tuned table.
  • 👉 M8 — Align It: Preference Tuning with DPO. Anvil learns to prefer the right call.
  • ⬜ M9 — The DPO Family: ORPO, KTO, SimPO. The rest of the preference family, and which to pick.
  • ⬜ M10 — Teach It to Reason: GRPO and Verifiable Rewards. RL with a checkable reward.
  • ⬜ M11–M15 — merging, quantization, serving, publication, and the capstone.

Before this module, Anvil imitates good tool calls (SFT) but still hesitates between plausible tools and rambles before the call. After it, Anvil prefers the valid, concise call — aligned with DPO. That DPO checkpoint becomes the starting point for the preference family (Module 9), for reasoning RL (Module 10, GRPO), and it gets re-evaluated at every stage and replayed in the capstone (Module 15).

What SFT can’t teach: imitation versus preference

Go back to what SFT actually does, because the gap lives there. Supervised fine-tuning minimizes a next-token cross-entropy loss over demonstrations: for each training example, the model is nudged to assign higher probability to the exact tokens of the target completion. For Anvil that target is a valid <tool_call>{...}</tool_call> string (or a clean refusal on negatives). Run enough of those and the model learns the format and the mapping — given this request and these tools, produce that call.

What SFT never does is show the model a bad answer labeled as bad. Every gradient it ever sees points toward “make this good answer more likely.” There is no contrast, so there is no notion of better than. And that’s exactly the capability Anvil is missing in those two failure cases from the intro:

  • Two plausible tools. SFT taught Anvil what a search_notes call looks like and what a search_emails call looks like. It never taught it that, for this prompt, the first is preferable to the second — because it only ever optimized one target at a time, never a ranking.
  • Verbose preamble. SFT’s gold targets are concise calls, so concise calls are likely. But a chatty completion that contains the right call is also fairly likely — and SFT has no mechanism to say “the concise one is better.” Both look like things it has seen.

So SFT teaches format; it does not teach preference. To teach preference you supply, for the same prompt, a chosen completion (the one you prefer) and a rejected completion (the one you don’t), and you optimize the model to assign more probability to chosen than to rejected. That comparison — the thing SFT structurally lacks — is what preference tuning adds. It’s how you shape behavior after SFT.

Side by side, the two objectives line up like this — and note that DPO does not replace SFT, it refines its checkpoint:

SFTDPO
ObjectiveReproduce the target tokens (next-token cross-entropy)Prefer chosen over rejected (a contrastive log-ratio loss)
DataDemonstrations — one gold completion per prompt(prompt, chosen, rejected) pairs — two completions per prompt
What it learnsThe format and the mapping (request + tools → call)The preference — “this completion is better than that one”
What it does NOT doNo notion of “better than” — never sees a labeled bad answerAdds no knowledge; doesn’t replace SFT — it refines the SFT checkpoint

The historical frame: RLHF-PPO

Preference tuning isn’t new. The technique that put it on the map is RLHF — Reinforcement Learning from Human Feedback — as deployed in InstructGPT (arXiv 2203.02155, the paper behind the original ChatGPT alignment). RLHF-PPO is a three-stage pipeline, and it’s worth seeing in full because DPO is best understood as the thing that collapses it:

  1. SFT. Fine-tune the base model on demonstrations (you already did this in Module 3).
  2. Reward model. Collect human preference comparisons (humans rank pairs of outputs), then train a separate model — the reward model — to predict which output a human would prefer. This is a second neural network, trained and stored.
  3. PPO. Optimize the SFT model (now the policy) with PPO — Proximal Policy Optimization, an online reinforcement-learning algorithm — to maximize the reward model’s score, while a KL penalty keeps the policy from drifting too far from the SFT reference.

It works, and it produced the first wave of aligned assistants. But look at what it costs. You train and hold in memory an extra reward model. You run an online RL loop — the policy generates rollouts, they get scored, gradients flow — which is notoriously unstable and hyperparameter-sensitive (reward hacking, mode collapse, KL blowups). It’s hard to reproduce and expensive to run. In 2026, presenting RLHF-PPO as “the default alignment method” is simply out of date — and in TRL, PPOTrainer has moved to trl.experimental.ppo to reflect exactly that (more on the freshness twist below). Here, PPO is historical context — never code you run.

⚠️ Common misconception: “DPO replaces SFT” / “DPO teaches the model new facts.”

Two false twins, and they’re the most important thing in this module to get right.

(a) “DPO replaces SFT” — no. DPO starts from the SFT checkpoint and refines it. Without a decent SFT first, there’s no baseline behavior to align — and, as you’ll see, the reference model that DPO needs literally is the SFT model, frozen. No SFT, no anchor, no DPO.

(b) “DPO teaches new facts or skills” — no. Preference tuning shapes behavior: it nudges the model toward preferring the better of the answers it already knows how to produce. It does not inject knowledge. If Anvil has never seen a tool schema for currency conversion, DPO won’t teach it one — that’s the job of data and SFT (and for live facts, RAG — see Module 1).

Say it as a single rule: DPO shapes behavior after SFT; it doesn’t add knowledge and it doesn’t replace SFT.

So the question DPO answers is: can we get RLHF’s preference-shaping without the reward model and without the online RL loop? Yes — and that’s the next section.

DPO: preference tuning without a reward model

DPO — Direct Preference Optimization (arXiv 2305.18290, NeurIPS 2023) — is the result that made preference tuning cheap and stable. Its central insight is a piece of math, and you don’t need the derivation to use it, only the intuition:

The reward model that RLHF trains has a known analytical solution in terms of the policy itself. So you can skip training the reward model and skip the online RL loop, and optimize the policy directly on the preference pairs — with a single classification-style loss.

That’s the “Direct” in Direct Preference Optimization: no intermediary. Concretely, DPO compares the policy’s log-probabilities on chosen and rejected to those of a frozen reference model, and pushes them apart in the right direction:

  • it increases log π(chosen) − log π_ref(chosen) — make the chosen answer more likely than the reference would, and
  • it decreases log π(rejected) − log π_ref(rejected) — make the rejected answer less likely than the reference would.

π is the policy (the model you’re training); π_ref is the reference (frozen). The loss is a logistic/classification loss over the difference of those two log-ratios, scaled by a hyperparameter beta. No reward model is ever instantiated; no completions are sampled during training. It’s an offline loss over a fixed dataset of pairs — which is why it trains as stably as SFT does.

The frozen reference model π_ref

The reference model is not an implementation detail — it’s the load-bearing part of why DPO is stable, so it gets its own paragraph. π_ref is, in the standard setup, the SFT model itself, frozen (its weights never receive gradients). It’s the anchor: the DPO loss only ever measures how far the trained policy has moved relative to this fixed point. Without it, “make chosen more likely” has no scale — the model could wander arbitrarily far from everything it learned in SFT, degenerating into nonsense that happens to score well on the contrast. The reference is the structural equivalent of PPO’s KL penalty, except it’s baked directly into the loss, with no RL machinery around it.

There’s a practical wrinkle that matters enormously on a free GPU. Naively, a reference model is a second full copy of the model in memory — fine on an A100, fatal on a 16 GB T4. But Anvil trains with LoRA/QLoRA (Module 5): the policy is a base model with a trainable adapter on top. In that case TRL doesn’t load a second model at all — it uses the base model with the adapter disabled as the reference (turn the adapter off for the reference forward pass, on for the policy forward pass). One set of base weights, two forward passes. That’s what keeps DPO on a 4B model comfortably inside a free T4. (Verify this behavior against the TRL version you install; as of trl 1.6.0, June 2026, PEFT runs use the adapter-disabled base as the reference automatically — you don’t pass a reference model at all.)

beta — the one knob that matters

beta (default 0.1 in DPOConfig — as of trl 1.6.0, June 2026) controls how far the policy is allowed to move from the reference. Read it as the conservative ↔ aggressive dial of alignment:

  • High beta (say 0.5) keeps the policy glued to the reference. Alignment is timid — the model barely changes, the reward margins barely move, and your eval delta is near zero.
  • Low beta (say 0.01) lets the policy move hard away from the reference. Alignment is aggressive — which can mean fast gains, but also degeneration, catastrophic forgetting, and over-optimizing the surface form of the preference instead of its intent.

The practical opinion: I start DPO around beta=0.1 with learning_rate=5e-6, and if the reward margins saturate within a handful of steps and the loss flatlines near zero, I lower the learning rate before I touch beta — that pattern is over-optimization, not a beta problem (see the pitfalls section). DPO over-corrects fast, which is why the learning rate sits one to two orders of magnitude below SFT’s.

Why DPO replaced RLHF-PPO

Here’s the head-to-head. This is the table to internalize — it’s the entire dpo vs rlhf argument:

RLHF-PPO (historical)DPO (2026 default)
ComponentsSFT model + a separate reward model + policy + frozen referencePolicy (the SFT model) + frozen reference. No reward model.
Online RL loop?Yes — the policy samples rollouts, they get scored onlineNo — offline supervised-style loss over fixed pairs
StabilityUnstable; sensitive to hyperparameters, prone to reward hackingStable; trains like supervised fine-tuning
Cost / memory≥ 3 models live + online generationPolicy + reference (and in LoRA, the reference is the adapter-off base — no extra copy)
ReproducibilityLow (RL variance)High (deterministic loss over a fixed dataset)
Default in 2026?No — historical; trl.experimental.ppoYes — top-level DPOTrainer

The pipeline difference is easiest to see as a diagram. SFT produces one checkpoint; DPO uses it twice — once as the trainable policy, once (frozen) as the reference — and the loss only updates the policy:

flowchart TB
    base["Qwen3-4B-Base"] --> sft["SFT (Module 3)"]
    sft --> ckpt["SFT checkpoint"]
    ckpt --> policy["policy π<br/>(trainable — gets DPO gradients)"]
    ckpt --> ref["reference π_ref<br/>(frozen — no grad)"]
    pairs["(prompt, chosen, rejected)"] --> policy
    pairs --> ref
    policy --> loss["DPO loss<br/>compares log π / log π_ref<br/>on chosen vs rejected"]
    ref --> loss
    loss -->|updates ONLY the policy| policy

    classDef frozen fill:#eee,stroke:#999,color:#333;
    class ref frozen;

The annotations to keep in your head: the reference is a frozen SFT anchor that replaces PPO’s reward model and KL penalty, and the whole thing is offline — no RL rollouts. That’s the entire reason DPO is the default: same preference-shaping outcome as RLHF, none of the moving parts.

Where the pairs come from: building preference data for tool-calling

DPO is only as good as its preference data, and this is the part most tutorials hand-wave. The data format itself is simple — DPO consumes examples shaped like:

{"prompt": "<user request + tools, rendered>", "chosen": "<preferred completion>", "rejected": "<dispreferred completion>"}

chosen and rejected share the same prompt; they differ only in the completion. As with SFT (Module 2), you don’t hand-roll the special tokens — the chat template renders the prompt, and you supply the prompt and the two completions as strings. The real question is: where do the pairs come from?

In practice, there are four sources, roughly ordered by cost and by how directly they reflect what you actually want:

  1. Human preferences. A person ranks two outputs (the original RLHF setup). The gold standard for capturing nuanced human taste — and the most expensive to collect at scale.
  2. AI feedback / LLM-as-judge (“RLAIF”). A stronger model ranks the pairs instead of a human. This is exactly the LLM-as-judge you wired up in Module 7, repurposed to label preferences rather than score a single output. Cheaper than humans; inherits the judge’s biases (position, verbosity, self-preference).
  3. On-policy sampling. Draw two completions from the model itself and keep the better one as chosen, the worse as rejected. Keeps the pairs close to the model’s own distribution — useful, but you need some way to decide which is better (often back to a judge or a verifier).
  4. Programmatic construction from a verifiable task. When your task has a checkable notion of correctness, you can build pairs mechanically: take a known-good answer as chosen, and degrade it into a plausible-but-wrong rejected. This is Anvil’s case — and it’s the cheapest reliable source, because the tool-calling task is exactly that verifiable.

Anvil’s recipe: degrade the gold call

Anvil already has gold tool calls (the fixture’s targets). So we synthesize rejected by degrading the gold chosen — one degradation mode per pair, rotated so the dataset stays balanced and the construction stays deterministic and pedagogical. The three modes mirror the failure cases we want to un-teach:

flowchart TB
    gold["gold tool call<br/>(from the fixture)"] --> chosen["chosen ✅<br/>the M7-valid call"]
    gold -->|degrade| r1["wrong tool name<br/>(get_weather → get_weather_v2)"]
    gold -->|degrade| r2["dropped arguments<br/>(arguments: {})"]
    gold -->|degrade| r3["chatty prose<br/>(preamble, no call)"]
    r1 --> rejected["rejected ❌"]
    r2 --> rejected
    r3 --> rejected
    chosen --> dpo["(prompt, chosen, rejected) → DPO"]
    rejected --> dpo

In the lab’s data.py the three degradations are _wrong_name (append _v2 to the function name — a call that looks right but targets a function that doesn’t exist), _drop_args (keep the name, empty the arguments — right tool, useless call), and _PROSE_REJECT (a chatty sentence with no tool call at all — the verbose preamble we want to kill). Each pair gets one mode, chosen by i % 3 over the rows. The key alignment property: chosen is exactly the completion eval.py would score as correct in Module 7 — so DPO optimizes directly toward the metric we already measure.

Pitfalls in the data itself

Three data traps, all at the “why” level:

  • rejected too easy. If your rejected is random noise (gibberish, an empty string), the model learns a trivial distinction — “valid-ish vs garbage” — not the real preference you care about. The degradations above are plausible failures on purpose.
  • chosen and rejected too close. If they barely differ, the preference signal is weak and DPO has little to push on. There’s a sweet spot between “trivially different” and “indistinguishable.”
  • Leakage. Do not build preference pairs from Module 7’s held-out eval split. If you train on contrastive versions of the very examples you grade on, your DPO-vs-SFT delta is contaminated. The preference pairs come from the train rows; the eval split stays untouched (reinforcing the splits/contamination discipline from Module 2).

Training Anvil with the stable DPOTrainer

Now the freshness twist that defines this module. DPOTrainer is stable in TRL v1 — you import it from the top-level trl, exactly like SFTTrainer. Its historical counterpart, PPOTrainer, has been moved to trl.experimental.ppo (legacy 🧪). That inversion — preference tuning promoted, online RLHF demoted — is the 2026 story in one import line. (As of trl 1.6.0, June 2026: DPOTrainer, SFTTrainer, GRPOTrainer, and KTOTrainer are top-level stable; ORPO, CPO, BCO, and PPO live in trl.experimental. Re-check the split against the version you install — it churns.) KTO/ORPO/SimPO themselves are Module 9; here we stay on stable DPO.

The canonical pattern

This is anvil/train_dpo.py, trimmed to the load-bearing lines. Note what’s there — and what’s deliberately not:

from trl import DPOTrainer

def train_dpo(*, tier: str = "full", use_lora: bool = True, data_limit: int | None = None,
              output_dir: str = "outputs/anvil-dpo", beta: float = 0.1, **arg_overrides):
    model, tokenizer = config.get_model_and_tokenizer(tier)        # base via the M5 factory (conceptually the SFT checkpoint)
    train_dataset = data.make_preference_dataset("train", tokenizer, limit=data_limit)
    args = config.dpo_args(output_dir=output_dir, beta=beta, **arg_overrides)
    peft_config = config.lora_config() if use_lora else None

    trainer = DPOTrainer(
        model=model,                    # the policy — conceptually the SFT checkpoint; in production load the saved SFT adapter first
        args=args,
        train_dataset=train_dataset,    # (prompt, chosen, rejected) triples
        processing_class=tokenizer,     # TRL v1 idiom — never tokenizer=
        peft_config=peft_config,        # LoRA adapter; the reference is the adapter-OFF base (auto)
    )
    trainer.train()
    trainer.save_model(output_dir)
    return trainer

And the config — anvil/config.dpo_args builds a DPOConfig:

from trl import DPOConfig

def dpo_args(output_dir: str = "outputs/anvil-dpo", **overrides):
    defaults = dict(
        output_dir=output_dir,
        beta=0.1,                       # distance from the reference — the conservative↔aggressive knob
        max_length=1024,                # NOT max_seq_length (removed; TypeError)
        learning_rate=5e-6,             # DPO sits 1-2 orders of magnitude below SFT
        per_device_train_batch_size=4,
        gradient_accumulation_steps=2,
        seed=0, report_to=[],
    )
    defaults.update(_precision_kwargs())   # bf16 on Ampere+, fp16 on a T4, neither on CPU
    defaults.update(overrides)
    return DPOConfig(**defaults)

Each line, at the “why” level rather than the “what”:

  • model= is conceptually the SFT checkpoint — DPO’s whole premise is to start from SFT behavior, never the raw base, because there’s no baseline behavior to align on a base model (the misconception box, made concrete). In the lab, get_model_and_tokenizer(tier="qlora") returns the base via the same frozen Module 5 loader (4-bit NF4 base, fresh adapter on top); loading your saved SFT adapter before DPO is the production step that makes the policy literally continue from SFT.
  • No ref_model argument. When the policy is a PEFT model, TRL builds the reference automatically by disabling the adapter — so you pass nothing. That’s the lightweight path that fits a T4. (In a full fine-tune you’d pass a frozen copy as ref_model; the LoRA path needs no second model.) Verify against your installed version.
  • processing_class=tokenizer — the TRL v1 idiom. tokenizer= was removed; passing it raises a TypeError.
  • max_length on DPOConfig — never max_seq_length (removed). (As of trl 1.6.0, DPOConfig exposes beta, loss_type, and max_length; there is no max_prompt_length field on it — don’t copy that from older tutorials.)
  • learning_rate=5e-6 — far below SFT’s 2e-4, because DPO’s contrastive gradient over-corrects quickly.

Reading a DPO run

Beyond the loss, watch the metrics TRL logs specifically for preference training: rewards/chosen, rewards/rejected, and especially the reward margin (chosen minus rejected) and rewards/accuracies (the fraction of pairs where the policy already prefers chosen). A healthy run shows the margin climbing steadily — the model preferring the good completion more and more, by a widening gap. A margin that explodes in a few steps while the loss flatlines at zero is the over-optimization tell: usually beta too low or learning rate too high.

The re-eval loop (the differentiator)

This is where the course’s spine — task-grounded evaluation — earns its keep. You re-run Module 7’s eval.py, unchanged, on the DPO model:

from anvil import eval as evaluate
rows = data.load_rows("eval")               # the held-out split, never trained on
results = evaluate.compare_base_vs_finetuned(base_model, dpo_model, tokenizer, rows)
# -> {"base": {...}, "finetuned": {...}}  valid_json / fn_acc / arg_match / abstention
forget = evaluate.anti_forgetting_probe(dpo_model, tokenizer)   # did alignment break general ability?

You produce the same base / SFT / DPO table that ships in Anvil’s model card (Module 14) — valid JSON, function-name accuracy, argument match on the held-out — plus the anti-forgetting probe that catches a model that got “more aligned” but lost general competence. The point is the discipline: you do not believe DPO helped; you measure it. (Report the delta as “on my run” — you’ll get your own numbers in the lab, and they depend on the run.)

Training-time pitfalls

  • beta too high → almost nothing changes. The policy stays glued to the reference; margins barely move; the eval delta is noise. Lower beta to let it move.
  • An absent or drifting reference → divergence or forgetting. The reference must be frozen; in the LoRA path TRL handles that for you, but if you ever pass your own reference, freeze it.
  • Reward hacking toward verbosity. DPO learns whatever distinguishes chosen from rejected. If your chosen answers are systematically longer, DPO may learn “prefer longer” — the opposite of what Anvil wants. For a tool-calling specialist we want concision, which is precisely why one of the three degradations makes rejected a verbose preamble: it pushes the gradient toward the short call.
  • Catastrophic forgetting. Aggressive alignment can erode general ability — which is exactly why the Module 7 anti-forgetting probe rides along on every DPO run.

Build it: align Anvil with DPO

Goal: align Anvil with DPO on a free Colab T4 — build (prompt, chosen, rejected) triples from the fixture, run train_dpo.py from the SFT checkpoint (QLoRA, frozen reference), then re-run Module 7’s eval.py to show the DPO-vs-SFT delta.

What you’ll see: the DPO loss falling and the reward margin (chosen − rejected) climbing in the TRL logs, then a base / SFT / DPO table from eval.py showing a DPO-vs-SFT gain (especially on concision / picking the right tool), with a stable anti-forgetting line.

The full, tested code lives in finetune-prod-labs/module-08. Its smoke tests run offline — no GPU, no network — by running a real 2-step DPOTrainer on a tiny Qwen3 fixture, so you can verify the plumbing before you spend a minute of GPU time.

Step 1 — Build the preference pairs. data.make_preference_pairs reads the positive rows of a split, renders the gold call as chosen, and degrades it into rejected (one mode per pair: wrong name, dropped args, or chatty prose). make_preference_dataset wraps that into a 🤗 Dataset with {prompt, chosen, rejected} columns — exactly what DPOTrainer wants. It builds from train, never from the held-out eval split.

Step 2 — Run the DPO training. From the SFT checkpoint, on a T4:

uv run --env-file .env python -m anvil.train_dpo

main() calls train_dpo(tier="qlora", use_lora=True): it loads the 4-bit base via the frozen Module 5 factory, builds the preference dataset, constructs DPOConfig (beta=0.1, max_length=1024, learning_rate=5e-6), and runs the stable DPOTrainer with processing_class=tokenizer and peft_config — no explicit reference model, because the adapter-off base is the reference. Watch the logs: loss down, rewards/margins up.

Step 3 — Re-evaluate with Module 7. Call eval.compare_base_vs_finetuned(...) on base / SFT / DPO over the held-out split, and eval.anti_forgetting_probe(...) on the DPO model. You get the same task table that goes in the model card, plus the forgetting glance. This is the proof — not a vibe that “it feels more aligned.”

You’ll see something like this (trimmed — your numbers will differ):

{'loss': 0.69, 'rewards/chosen': 0.00, 'rewards/rejected': 0.00, 'rewards/margins': 0.00, ...}
{'loss': 0.42, 'rewards/chosen': 0.31, 'rewards/rejected': -0.58, 'rewards/margins': 0.89, ...}
{'loss': 0.28, 'rewards/chosen': 0.52, 'rewards/rejected': -1.04, 'rewards/margins': 1.56, ...}

base / SFT / DPO on held-out (n=200):
                valid_json   fn_acc   arg_match
  base              0.06       0.09      0.04
  SFT               0.93       0.88      0.81
  DPO               0.95       0.91      0.86
  anti-forgetting probe (DPO): 1.00

The margin climbing from 0.00 toward ~1.5 is DPO learning the preference; the small but real DPO-vs-SFT bump (and a stable forgetting probe) is the alignment paying off without breaking anything.

Step 4 — The smoke test. The offline suite (module-08/tests/smoke_test.py) builds preference triples on the tiny fixture and runs a real 2-step DPOTrainer — verifying that chosen != rejected, that the verbose-prose degradation mode appears across the rows, that dpo_args carries beta and max_length, and that the trainer logs preference metrics. It needs no GPU and no network:

uv run pytest module-08/tests/

What this lab does NOT do, on purpose. No ORPO / KTO / SimPO — those are Module 9 (we stay on stable DPO; one-sentence teaser only). ORPO and SimPO are trl.experimental in TRL v1; KTO is stable top-level, like DPO. No GRPO and no reward_fn — DPO is offline on fixed pairs, not RL with a verifiable reward; that distinction is Module 10. No PPO — PPOTrainer is trl.experimental.ppo, historical context, never executed here. No merging, quantization, serving, or Hub push (Modules 11–14). The real QLoRA run needs a CUDA GPU (free Colab/Kaggle T4); the smoke test runs on CPU.

Try it yourself:

  1. Sweep beta (try 0.05, 0.1, 0.5) and watch both the reward margins and the Module 7 table. Find the trade-off between alignment strength and forgetting — too high barely moves, too low risks degeneration.
  2. Change the rejected strategy — make every pair a verbose-prose rejection, or every pair a wrong-name rejection — and see which degradation buys the biggest concision/accuracy gain at eval. (Hint: the prose mode is the one that fights verbosity directly.)

In production

What changes outside the lab? The single most important shift: the preference data is the bottleneck, not the trainer. Running DPOTrainer is the easy part. Manufacturing reliable, varied pairs — human-labeled, judge-labeled, on-policy, or programmatically degraded — is where the cost and the risk live. Treat that dataset as a first-class artifact: version it like code (decision F14 — reproducibility), because your alignment is only as reproducible as the pairs that produced it.

Second: always measure DPO-vs-SFT on a held-out split. Alignment can degrade the task — a model that learned to prefer something subtly wrong, or that over-optimized verbosity, can score worse than the SFT model it came from. That’s why Module 7’s eval runs after every alignment stage, not just once. Watch catastrophic forgetting too: a model that’s “more aligned” but has lost its general competence is a bad trade.

On cost and hardware (decision F6): a QLoRA DPO run on a 4B model fits a free T4 — it’s offline, no RL rollouts, lighter than GRPO. A longer or full-precision run goes on a rented GPU; ballpark $1.19–$2.69/hour for an A100 depending on provider (as of June 2026 — re-check, and see Module 1’s GPU cost notes). My standing opinion: reach for DPO first precisely because it’s stable and data-cheap; reach for RL (GRPO, Module 10) only when you have a verifiable reward and you’re trying to push reasoning — not for ordinary preference alignment.

And the one-line teaser for what’s next: DPO has a whole family — ORPO (folds preference into a single stage, no separate SFT), KTO (learns from binary good/bad labels instead of pairs), and SimPO (drops the reference model entirely) — and which one to pick depends on the data you have. That’s Module 9. ORPO and SimPO are trl.experimental in TRL v1; KTO is stable top-level, like the DPO default you just used.

Concept check

Why this matters. These five questions exercise the core of T8’s DPO half: what SFT can’t teach, why DPO replaced RLHF-PPO, where preference pairs come from, the role of the frozen reference model, and how to read and tune beta. The DPO checkpoint you built here feeds Module 9 (the family), Module 10 (GRPO), and the Module 15 capstone — so getting these straight pays off repeatedly. The questions stay inside what this module taught.

  1. A teammate says: “Our SFT model already emits valid tool calls, so DPO will replace the SFT and also teach it new APIs we never trained on.” What’s wrong, and what does DPO actually do?

    • A. Nothing’s wrong — DPO is a stronger SFT that also adds knowledge.
    • B. DPO doesn’t replace SFT; it starts from the SFT checkpoint and shapes behavior — preferring among answers the model already knows how to produce. It does not add new knowledge.
    • C. DPO replaces SFT but doesn’t add knowledge.
    • D. DPO adds knowledge but runs before SFT.

  2. You want to DPO-tune Anvil, but you only have known-good (“gold”) tool calls — no human preference labels. How do you build (prompt, chosen, rejected)?

    • A. You can’t do DPO without human labels.
    • B. Use the gold call as chosen and degrade it programmatically (wrong name / dropped args / verbose prose) for rejected; other sources include human labels, an LLM-as-judge, and on-policy sampling.
    • C. Use the gold call as rejected and a random string as chosen.
    • D. Use two copies of the gold call as chosen and rejected.

  3. Why does DPO need a frozen reference model, and what is it for a LoRA/QLoRA run?

    • A. It’s an optional speedup; you can always omit it.
    • B. It’s a second reward model trained on preferences.
    • C. It’s the frozen anchor (the SFT model) that replaces PPO’s reward model + KL penalty and keeps the policy from drifting; in a PEFT run it’s the base model with the adapter disabled — no second model is loaded.
    • D. It’s the raw base model before any SFT.

  4. A colleague proposes re-implementing PPO for alignment “because that’s how RLHF works.” Why is DPO the better default in 2026, and what’s the freshness gotcha?

    • A. PPO is still the default; DPO is experimental.
    • B. DPO needs no separate reward model, no online RL loop, is more stable, reproducible, and cheaper; and PPOTrainer now lives in trl.experimental.ppo while DPOTrainer is top-level stable.
    • C. They’re equivalent; pick either.
    • D. DPO is only for chatbots, PPO for tool-calling.

  5. A few steps into DPO, the loss collapses toward zero and the reward margins explode. What’s happening, and what do you change first?

    • A. Training converged perfectly; stop.
    • B. beta is too high; lower it.
    • C. Over-optimization (beta too low or learning rate too high). Lower the learning rate first; a too-high beta, by contrast, would cause almost no change at all.
    • D. The reference model is too strong; remove it.

Answers.

  1. B. DPO refines the SFT checkpoint and shapes behavior — it nudges the model to prefer the better of the outputs it already produces. It neither replaces SFT (it depends on it; the reference is the SFT model) nor injects knowledge (that’s data + SFT, or RAG for live facts).
  2. B. Anvil’s task is verifiable, so you can manufacture pairs: gold = chosen, a controlled degradation = rejected. The other real sources are human preferences, an LLM-as-judge (RLAIF), and on-policy sampling. You do not need human labels to start.
  3. C. The reference is the frozen anchor — the SFT model, which plays the role of PPO’s reward model + KL penalty inside the loss, keeping the policy from drifting arbitrarily. In a LoRA/QLoRA run, TRL uses the adapter-disabled base as the reference, so no second full model is loaded (which is what fits a T4).
  4. B. DPO drops the reward model and the online RL loop, trains stably like supervised fine-tuning, and is cheaper and more reproducible. The freshness gotcha: in TRL v1 PPOTrainer is trl.experimental.ppo (legacy) while DPOTrainer is stable top-level — PPO is not the 2026 default. (As of trl 1.6.0, June 2026.)
  5. C. Loss → 0 with exploding margins is over-optimization: the model is trivially separating chosen from rejected, usually because beta is too low or the learning rate too high. Lower the learning rate first. A too-high beta, the opposite failure, glues the policy to the reference and produces almost no change.

Common pitfalls.

  • “DPO replaces SFT / teaches new facts.” The central conceptual trap. DPO starts from the SFT checkpoint and shapes behavior; it adds no knowledge (facts come from data/SFT, or RAG — Module 1). Without a good SFT first, there’s nothing to align.
  • Treating RLHF-PPO as “the default,” or importing PPOTrainer from trl. Wrong in 2026 — DPO is the default, and PPOTrainer moved to trl.experimental.ppo. PPO here is historical context, never code you run. (As of trl 1.6.0, June 2026.)
  • Building rejected at random, or building pairs on the held-out split. A trivial rejected gives a useless signal; building preference pairs on the eval split is leakage that contaminates your DPO-vs-SFT delta. The preference pairs come from train; the eval split stays untouched.

Key takeaways

  • SFT imitates; it has no notion of “better than.” It minimizes next-token loss on good answers and never sees a labeled bad one — so it can’t learn to prefer the concise, correct call. Preference tuning shapes behavior after SFT.
  • DPO ≠ replacing SFT, and DPO ≠ adding knowledge. It refines the SFT checkpoint to prefer among answers the model already produces; the reference model literally is the frozen SFT.
  • DPO (arXiv 2305.18290) replaced RLHF-PPO (InstructGPT, arXiv 2203.02155) because it needs no separate reward model and no online RL loop: one stable, reproducible, cheaper supervised-style loss.
  • The data is (prompt, chosen, rejected) triples. For Anvil: gold call = chosen, controlled degradation (wrong name / dropped args / verbose prose) = rejected — aligned with the Module 7 metric. Other sources: human, LLM-judge, on-policy.
  • The frozen reference model anchors the policy (= the SFT model, frozen; in LoRA, no second model — the adapter-off base is the reference), and beta is the conservative↔aggressive knob — too high = inert, too low = degeneration.
  • DPOTrainer is stable in TRL v1 (PPOTrainer is trl.experimental.ppo); use the v1 idioms — processing_class, max_length on DPOConfig — as of trl 1.6.0, June 2026.
  • Always re-evaluate (Module 7): DPO-vs-SFT plus an anti-forgetting glance. Alignment can degrade the task — measure it, don’t assume it.

What’s next

Anvil prefers the right tool call now. Module 9 meets the rest of the family — ORPO (no separate SFT stage), KTO (binary good/bad labels instead of pairs), and SimPO (no reference model at all) — and gives you a decision tree for which one your data calls for. ORPO and SimPO are trl.experimental in TRL v1; KTO is stable top-level, alongside the DPO foundation they build on.

Want the full fine-tuning concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · ← Module 7: Is It Any Better? Task-Grounded Evaluation · Module 9: The DPO Family → · Lab code for this module

References