aifromzero.dev

The DPO Family: ORPO, KTO, SimPO — and When to Use Each (Fine-Tuning in Production, Module 9)

Module 9 of 15 14 min read Lab code ↗

This is Module 9 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Start at Module 1 or browse the full syllabus.

You aligned Anvil. You also paid three taxes you never questioned.

In Module 8 you aligned Anvil with DPO, and it worked: Anvil now prefers the precise <tool_call> over a plausible-looking failure, it stays concise, and eval.py proves the lift over the base model. Good. Ship it.

But look at the pipeline you actually ran, because it quietly charged you three taxes. First, it took two stages — you supervise-fine-tuned Anvil in Module 3, then ran a separate DPO pass on top. Second, DPO kept a reference model resident the whole run — a frozen second copy of the network sitting in memory just to anchor the policy. Third, it demanded preference pairs: for every prompt, a chosen and a matched rejected. We manufactured those pairs by degrading the gold call, but in the real world the feedback you collect almost never arrives pre-paired. It arrives as a thumbs-up on one answer. A “this call was wrong” flag in a log. A tool that either succeeded or threw.

So three honest questions fall out: can you align without a separate SFT pass? Without a reference model burning VRAM? Without paired data you don’t have? The answer to each is a different member of the DPO family — ORPO, KTO, and SimPO. This module is not “here are newer, better methods.” It’s the opposite: none of them beats DPO in general. The skill is reading the data you actually have and the constraints you actually face, then picking. You’ll re-align Anvil with one alternative and put it head-to-head with your DPO baseline — and walk away with a decision tree you can defend in a design review.

In this module

You’ll learn how to:

  1. Map the DPO family — DPO, ORPO, KTO, SimPO — on the three axes that actually drive the choice: does it need a reference model, what data format does it eat (pairwise vs binary), and is it single-stage (folds SFT in) or a separate alignment pass.
  2. Explain what each method changes versus DPO and why: ORPO (arXiv 2403.07691) folds SFT and alignment into one monolithic loss with an odds-ratio penalty and no reference model; KTO (2402.01306) trains on binary good/bad labels instead of pairs; SimPO (2405.14734) is reference-free and length-normalized with a target margin.
  3. Choose a method from the data you have, with a decision tree: pairwise data → DPO/SimPO; binary good/bad signals → KTO; want a single-stage run with no reference model → ORPO.
  4. Run one alternative on Anvil — ORPO (from the base, no separate SFT) or KTO (binary data derived from the pairs) — through one method switch in train_dpo.py, and compare it to the Module-8 DPO baseline with the Module-7 evaluation.
  5. Recognize the freshness trap: in TRL 1.6.0, only DPOTrainer (and GRPOTrainer) sit on the rock-solid stable API; ORPO and SimPO/CPO live in trl.experimental, and even KTO, though importable from the top level, is flagged as still-migrating with “API changes may occur.” Pin your version.

You’ll build: Anvil’s train_dpo.py grows a method switch — dpo (your Module-8 baseline), plus orpo, kto, and a simpo flavor. You’ll re-align Anvil with ONE alternative — ORPO straight from the base with no separate SFT, or KTO on binary good/bad data — and put it head-to-head with your DPO baseline using the Module-7 task-grounded eval. By the end you own a data→method decision tree and a comparison table you can defend.

Theory areas covered:

  • T8 — Preference tuning: DPO, ORPO, KTO, SimPO — 11% of the course (co-owned with Module 8). Module 8 built the DPO part of T8; this module builds the family part — ORPO, KTO, SimPO, the experimental-status freshness trap, and the data→method decision tree. T8 is reinforced again in the Module-15 capstone.

This module stands on T8/DPO from Module 8 (pairs, DPOTrainer, the reference model, β — recalled here, not re-taught) and reuses T7 (evaluation, Module 7) unchanged to compare the methods. Nothing new is invented for measurement: the same eval.py that judged DPO judges ORPO and KTO.

Prerequisites: Modules 1–8, especially M8 (DPO: pairs, DPOTrainer, the reference model, β) and M7 (eval.py, base-vs-fine-tuned). You need the installed stack, config.py’s get_model_and_tokenizer(tier), and the dataset fixture with its preference pairs. For the lab: a free Colab/Kaggle T4 (16 GB) is the default path — aligning a 4B model in QLoRA fits just as it did in Module 8 — with HF_TOKEN set (Module 1).

Where you are

You’re nine modules into forging Anvil. Here’s the map; from Module 1 on, every module marks your place with ✅ (done), 👉 (you’re here), and ⬜ (ahead):

  • ✅ M1 — When (Not) to Fine-Tune. Baseline: the base model fails the tool-call format.
  • ✅ M2 — Data Is the Model. anvil/data.py, chat templates, the tool-calling fixture.
  • ✅ M3 — Your First Fine-Tune: SFT. train_sft.py, Anvil v0.
  • ✅ M4 — LoRA. Train 1% of the weights.
  • ✅ M5 — QLoRA. Anvil fits a free 16 GB T4.
  • ✅ M6 — Unsloth & Axolotl. Faster runs, config-as-code.
  • ✅ M7 — Task-Grounded Evaluation. eval.py, base-vs-fine-tuned.
  • ✅ M8 — Align It: DPO. train_dpo.py, preference pairs, the reference model, β.
  • 👉 M9 — The DPO Family: ORPO, KTO, SimPO. A method switch; re-align via one alternative; compare to the DPO baseline.
  • ⬜ M10 — Teach It to Reason: GRPO and Verifiable Rewards. Reward becomes computed, not preferred.
  • ⬜ M11 — Model Merging with mergekit.
  • ⬜ M12 — Shrink It: GGUF, AWQ, GPTQ.
  • ⬜ M13 — Serve It: vLLM, Ollama, TGI.
  • ⬜ M14 — Ship It: Publishing & Model Cards.
  • ⬜ M15 — Capstone: Anvil, End to End.

Anvil’s state before this module: aligned with DPO — two stages (SFT then DPO), a reference model in memory, preference pairs as input. After this module: it knows the whole DPO family, you’ve re-aligned it with one alternative (ORPO with no separate SFT, or KTO on binary data) and benchmarked it against the DPO baseline. This module closes the preference-tuning area (T8 = M8 + M9). Module 10 leaves preferences behind for verifiable reinforcement learning — GRPO and RLVR — where “preference” stops being a human signal and becomes a computed reward.

The frictions DPO left on the table

Three sentences of recall, no more, because Module 8 owns DPO. DPO (Direct Preference Optimization, arXiv 2305.18290) trains directly on preference pairs — (prompt, chosen, rejected) triples — to make the chosen completion more likely than the rejected one, relative to a frozen reference model, with a temperature β that controls how far you drift from that reference. It needs a prior SFT pass (you align on top of learned behavior), and it replaced RLHF-PPO as the 2026 default because it skips the separate reward model and the online RL loop. That’s the baseline. It’s stable, it’s well understood, and it’s the right first reach.

But notice what that one paragraph assumes. It assumes you can pay for three things, and the whole DPO family is organized around relaxing one of them each:

  1. Two stages — SFT then a separate alignment pass. Can you fold them into one run? → ORPO.
  2. A resident reference model — roughly a second copy of the network in memory for the whole run. Can you drop it? → ORPO (no reference) and SimPO (reference-free).
  3. Paired data — a matched chosen and rejected for every prompt. What if all you have is binary signals, a thumbs-up or thumbs-down on single completions? → KTO.

Here is the thesis of the module, in one sentence: the family does not “beat” DPO in general — each member makes a different trade, and it’s your data and your constraints that decide which trade is free for you. Hold that. Every section below is a variation on it.

The family, member by member: ORPO, KTO, SimPO

Each method gets the same four-beat treatment: a one-line definition, why it exists, what it requires and what it saves, and the exact TRL object plus its status as of trl 1.6.0, June 2026. Read them as three different answers to “which tax can I stop paying?”

ORPO — folds SFT and alignment into one loss

ORPO (Odds Ratio Preference Optimization, arXiv 2403.07691) is single-stage: one loss combines the ordinary SFT objective (the log-likelihood of the chosen completion) and a preference term built on the odds ratio — the ratio of the odds of generating chosen versus rejected. There is no separate SFT pass (the SFT objective is inside the loss) and no reference model (the anchor comes from that SFT term itself). It still eats pairwise (prompt, chosen, rejected) data, exactly like DPO.

Why it exists. The ORPO paper’s observation is sharp: plain SFT doesn’t only pull the model toward good answers — it also, as a side effect, raises the probability of unpreferred answers that happen to share tokens and structure with the good ones. ORPO bolts a penalty onto the SFT loss that directly suppresses the odds of the rejected completion while you’re doing SFT. That’s the “monolithic” part: instead of “learn the behavior, then later teach a preference,” it teaches the preference during the behavior learning, in a single pass, from the base model. What you save is concrete — one whole training stage and the memory of a reference model. As of trl 1.6.0 (June 2026): from trl.experimental.orpo import ORPOConfig, ORPOTrainer — experimental.

KTO — learns from binary thumbs-up/thumbs-down

KTO (Kahneman-Tversky Optimization, arXiv 2402.01306) eats a completely different data shape. Each example is (prompt, completion, label ∈ {good, bad}) — a single completion with a binary label. No matched pairs. It keeps a reference model, like DPO.

Why it exists. The name is the tell: KTO borrows from prospect theory (Kahneman and Tversky), which models how people value gains and losses relative to a reference point rather than in absolute terms. Instead of asking “is chosen more probable than rejected?”, KTO models the utility of a single completion — how desirable or undesirable it is against a reference — and pushes good completions up and bad ones down. The decisive practical consequence: real-world feedback is usually binary. Thumbs up/down. Accept/reject in a review queue. A tool call that verifiably succeeded or failed. That signal is the natural KTO format. Faking pairs out of it — pretending a random good and a random bad are a matched pair — throws away information and can inject noise. KTO consumes the binary signal directly. As of trl 1.6.0 (June 2026): from trl import KTOConfig, KTOTrainer works from the top level — but see the misconception box: TRL itself warns this path is still migrating.

SimPO — reference-free and length-normalized

SimPO (Simple Preference Optimization, arXiv 2405.14734) is reference-free (no second model, like ORPO) and length-normalized. The implicit reward it optimizes is the average log-probability per token (the sequence log-prob divided by its length), and the objective targets a margin (a gamma you set) between chosen and rejected.

Why it exists. DPO has a known length bias: because its implicit reward is a sum of log-probs, longer completions can accrue more reward, nudging the model toward verbosity. SimPO normalizes by length to neutralize that, and drops the reference model to lighten the run. For a tool-calling specialist that should emit one tight call, length control is a feature you might actually want. The TRL subtlety that trips people up: there is no SimPOTrainer. In TRL, SimPO is a loss variant of CPO. As of trl 1.6.0 (June 2026), you build it with from trl.experimental.cpo import CPOConfig, CPOTrainer and pass CPOConfig(loss_type="simpo", ...) with the margin field (simpo_gamma). It eats pairwise data like DPO. Do not invent a SimPOTrainer — you’ll get an ImportError.

⚠️ Common misconception: “These are newer than DPO, so they’re better — I should just use the latest one.”

Wrong, twice over.

(a) Newer is not better. None of ORPO, KTO, or SimPO beats DPO universally. Which one wins depends on your dataset, your model, and your tuning — the benchmarks in the papers are results from their setups, not promises about yours. You don’t pick by publication date; you pick by the shape of your data and your constraints (do you have pairs or binary signals? can you afford a reference model? do you want one stage?).

(b) Newer is not the same as stable. In trl 1.6.0 (June 2026), DPOTrainer and GRPOTrainer are stable. ORPO and CPO (and therefore SimPO) live in trl.experimental — importable and usable, but their API can shift between releases without a long deprecation window. And KTO is a deliberate in-between: you can import it from the top level (from trl import KTOTrainer), but TRL emits a FutureWarning that it’s being moved under trl.experimental.kto and that “API changes may occur.” So treat KTO/ORPO/SimPO as code you pin hard and re-verify on every upgrade. The freshness law of this course is “teach the current real idioms” — and the current real idiom is: only DPO/GRPO are bedrock; the rest you pin and watch.

The decision: data → method

The guiding principle is the one sentence to take from this whole module: you don’t choose a method, you choose from your data and your constraints. Two entry questions get you most of the way:

  1. What shape are my preference signals — matched pairs (chosen, rejected), or binary labels (👍/👎) on single completions?
  2. Do I already have an SFT, and can I afford a reference model in memory?

Here’s the tree those questions trace out:

graph TD
    Q0["What preference data do you have?"]
    Q0 -->|"pairwise (chosen, rejected)"| A{Constraints?}
    Q0 -->|"binary good/bad labels (no pairs)"| KTO["<b>KTO</b><br/>ref model: yes · single-stage: no<br/>eats binary {prompt, completion, label}"]

    A -->|"already SFT'd, want the stable default"| DPO["<b>DPO</b><br/>ref model: yes · single-stage: no<br/>the safe default — only stable trainer"]
    A -->|"want ONE run from the base, drop the ref"| ORPO["<b>ORPO</b><br/>ref model: no · single-stage: yes<br/>SFT + odds-ratio in one loss"]
    A -->|"want reference-free + length control"| SimPO["<b>SimPO</b><br/>ref model: no · single-stage: no<br/>length-normalized, target margin"]

    note["When in doubt: DPO.<br/>It's the only fully-stable trainer here."]

And the table to internalize. This is the heart of the module:

The DPO family at a glance

DPOORPOKTOSimPO
Data formatpairwise (chosen, rejected)pairwise (chosen, rejected)binary (completion, good/bad)pairwise (chosen, rejected)
Reference model?yesnoyesno
Single-stage (folds SFT)?no (needs prior SFT)yes (monolithic)nono
Key ideadirect on the implicit reward from prefsSFT + odds-ratio penalty in one lossutility of good/bad (prospect theory)length-normalized, reference-free, target margin
TRL status (trl 1.6.0, June 2026)stable — from trl import DPOTrainerexperimental — trl.experimental.orpotop-level from trl import KTOTrainer, but migrating (warns)via trl.experimental.cpo, CPOConfig(loss_type="simpo") — no SimPOTrainer
When to usedefault; you have pairs and a prior SFTone run from the base, no ref, tight VRAMyour real feedback is binary 👍/👎reference-free + length control on pairs

Here’s how I actually decide in production, with the numbers attached. I reach for DPO first — it’s the only stable trainer and the best understood, so it’s the lowest-risk lift. I switch to KTO the moment my feedback is genuinely binary (thumbs, accept/reject logs, verified tool successes), because faking pairs out of binary data throws away signal and can hurt. I try ORPO when I want to skip the separate SFT pass and the reference-model memory — handy on a tight free-GPU budget where every gigabyte counts — but I pin trl hard and add a test that breaks loudly if the experimental import path moves. And SimPO I treat as a length-control experiment: worth a run when DPO’s output is creeping verbose, never my first reach.

One thing no method buys you: a free pass on evaluation. Whatever you pick, you re-measure with eval.py (valid JSON, function-name accuracy, argument match, base-vs-fine-tuned). “Newer method” never excuses skipping the Module-7 eval on your task. A method that looks great on a paper’s benchmark can be flat or worse on Anvil’s tool-calling task — and the only way you find out is to measure.

Running an alternative on Anvil: ORPO (no SFT) or KTO (binary)

Per the frozen contract (06-FIL-ROUGE-SPEC §2), everything lives in train_dpo.py, which “grows ORPO/KTO/SimPO in M9.” We add one dispatcher, train_preference, that takes a method argument and routes to the right trainer. The dpo branch is your Module-8 baseline, untouched. Here’s the shape of the four-way switch — the full file is in the lab repo:

# anvil/train_dpo.py — method ∈ {"dpo", "orpo", "kto", "simpo"}
PREFERENCE_METHODS = ("dpo", "orpo", "kto", "simpo")

def train_preference(method: str = "dpo", *, tier: str = "full", use_lora: bool = True,
                     data_limit: int | None = None, output_dir: str | None = None,
                     **arg_overrides):
    method = method.lower()
    if method not in PREFERENCE_METHODS:
        raise ValueError(f"method must be one of {PREFERENCE_METHODS}, got {method!r}")
    if method == "dpo":
        return train_dpo(tier=tier, use_lora=use_lora, data_limit=data_limit,
                         output_dir=output_dir or "outputs/anvil-dpo", **arg_overrides)
    # ... ORPO / KTO / SimPO branches below

The freshness trap, hammered. The experimental trainers do not come from trl directly. As of trl 1.6.0 (June 2026):

from trl import DPOTrainer, DPOConfig                       # stable, top-level
from trl.experimental.orpo import ORPOTrainer, ORPOConfig   # experimental — verify path on your pin
from trl import KTOTrainer, KTOConfig                        # top-level today, but TRL warns it's migrating
from trl.experimental.cpo import CPOTrainer, CPOConfig       # SimPO = CPO with loss_type="simpo"

Set TRL_EXPERIMENTAL_SILENCE=1 in the environment to mute the import warnings once you’ve acknowledged them. Never present ORPOTrainer or CPOTrainer as stable, and re-check the exact import path against the trl version you actually installed — the paper-method-to-TRL-object mapping is the single most version-fragile fact in this module.

The four branches share a common config block (the same TRL v1 idioms you’ve used since Module 3 — processing_class, max_length, hyperparameters on the *Config), then each constructs its own trainer:

if method == "orpo":
    from trl.experimental.orpo import ORPOConfig, ORPOTrainer
    dataset = data.make_preference_dataset("train", tokenizer, limit=data_limit)
    args, Trainer = ORPOConfig(beta=0.1, **common), ORPOTrainer
elif method == "kto":
    from trl import KTOConfig, KTOTrainer
    dataset = data.make_kto_dataset("train", tokenizer, limit=data_limit)   # binary, not pairs
    args, Trainer = KTOConfig(beta=0.1, **common), KTOTrainer
else:  # simpo, implemented as CPO with loss_type="simpo"
    from trl.experimental.cpo import CPOConfig, CPOTrainer
    dataset = data.make_preference_dataset("train", tokenizer, limit=data_limit)
    args = CPOConfig(loss_type="simpo", cpo_alpha=0.0, simpo_gamma=0.5, **common)
    Trainer = CPOTrainer

trainer = Trainer(model=model, args=args, train_dataset=dataset,
                  processing_class=tokenizer, peft_config=peft_config)   # NOT tokenizer=

Three details that are easy to get wrong, and that the lab gets right:

Path A — ORPO with no separate SFT. ORPO starts from the base model (not Anvil’s SFT checkpoint), because the ORPO loss contains the SFT objective. There’s no ref_model passed and no prior SFT run — that’s the entire point of “monolithic / single-stage / reference-free.” It eats the same make_preference_dataset pairs as DPO. The odds-ratio weight is the beta field here; verify the exact name on your pin if you bump versions.

Path B — KTO on binary data. KTO does not eat pairs — it needs the binary {prompt, completion, label} schema, so we derive it. In anvil/data.py, make_kto_dataset splits each preference pair into two rows:

def make_kto_dataset(split, tokenizer, *, limit=None):
    """KTO wants *unpaired binary* data: {prompt, completion, label}."""
    pairs = make_preference_pairs(split, tokenizer, limit=limit)
    rows = []
    for p in pairs:
        rows.append({"prompt": p["prompt"], "completion": p["chosen"], "label": True})
        rows.append({"prompt": p["prompt"], "completion": p["rejected"], "label": False})
    return Dataset.from_list(rows)

Be honest about what this is: in the real world you’d have collected this binary signal directly (👍/👎 on tool calls). Here we synthesize it from Module 8’s pairs purely for the demo, so you can run KTO without a labeling team. And KTO keeps a reference model (like DPO) — say it out loud, because it’s the one thing people assume KTO drops and it doesn’t.

Comparing to the DPO baseline (reuse Module 7). After the run, you re-run eval.py — the same frozen compare_base_vs_finetuned and task metrics — on DPO-Anvil (Module 8) versus your ORPO-or-KTO-Anvil. Now the honest part: on a small demo run, the gaps are small and noisy. The pedagogical payoff here is the mechanism and the choice, not a definitive leaderboard. Don’t oversell a winner; report what your run shows and move on.

graph LR
    Base["Qwen3-4B-Base"]

    Base --> SFT["SFT (M3)"]
    SFT --> DPOp["DPO (M8)<br/>pairs + ref model<br/>2 stages"]
    Base --> ORPOp["ORPO<br/>one stage · no ref<br/>pairs"]
    Base --> KTOp["KTO<br/>binary labels + ref model<br/>fresh adapter on base"]

    DPOp --> Eval["eval.py (M7)"]
    ORPOp --> Eval
    KTOp --> Eval
    Eval --> Table["comparison table<br/>valid_json · fn_acc · arg_match"]

This lab deliberately does NOT do a few things, and it’s worth naming them. No GRPO / RLVR — that’s Module 10; here “preference” is still a human signal (a pair or a binary label), not a computed reward. No presenting ORPO/KTO/SimPO as stable — experimental imports, status flagged, version pinned. No SimPOTrainer — it doesn’t exist; SimPO is CPOConfig(loss_type="simpo"). No re-teaching DPO — three sentences of recall, that’s all. And no inference quantization, merging, serving, or publishing — those are Modules 11–14.

Build it: re-align Anvil with one alternative

Goal: extend train_dpo.py with a method switch, re-align Anvil with ONE alternative (ORPO from the base, or KTO on binary data), and compare it to the Module-8 DPO baseline with eval.py.

What you’ll see: the loss for your chosen method goes down on a free T4, the run finishes in QLoRA, and eval.py prints a small comparison table — DPO-Anvil versus your new variant — with the task metrics side by side.

The full, tested code lives in finetune-prod-labs/module-09. The offline smoke test runs the real ORPO/KTO/SimPO trainers for 2 steps on a tiny model — no GPU, no network — so you can verify the plumbing before spending a minute of GPU time.

Step 1 — Open the notebook on Colab/Kaggle (T4 runtime). Install the pinned stack (trl==1.6.0, peft==0.19.1, transformers==5.12.0, datasets==5.0.0, accelerate==1.14.0, bitsandbytes==0.49.2 for the 4-bit path), put HF_TOKEN in the environment (never hard-coded — os.environ["HF_TOKEN"]), and confirm the GPU is visible.

Step 2 — Pick your alternative and run it. Both load the base via the M5 factory and attach a fresh LoRA adapter — ORPO is single-stage and reference-free, KTO keeps a reference; both run in QLoRA on the T4:

from anvil.train_dpo import train_preference

# Path A: ORPO — one stage, from the base, no reference model
trainer = train_preference("orpo", tier="qlora", use_lora=True, data_limit=2000)

# Path B: KTO — binary labels, keeps a reference (trains a fresh adapter on the base, same as ORPO)
# trainer = train_preference("kto", tier="qlora", use_lora=True, data_limit=2000)

Watch the loss descend, exactly as it did for DPO in Module 8. The adapter saves to outputs/anvil-orpo (or -kto).

Step 3 — (KTO only) understand the binary derivation. Before you train KTO, inspect what make_kto_dataset produced: each preference pair became two rows, the chosen labelled True and the rejected labelled False. This is the moment to internalize that in production you’d collect this binary signal, not derive it.

Step 4 — Compare to the DPO baseline (reuse Module 7). Run eval.py’s compare_base_vs_finetuned on your Module-8 DPO adapter and your new variant over the held-out eval split, and print the table. Read it honestly: on this small demo the differences are small and noisy. You’re confirming the mechanism works end to end, not crowning a champion.

Step 5 — Run the offline smoke test. From the repo root:

uv run pytest module-09/tests/
......                                                            [100%]
6 passed in ~16s

The suite runs a real 2-step run of orpo, kto, and simpo on the tiny offline model, checks that the binary KTO dataset has exactly {prompt, completion, label} columns with both True and False present, and asserts a bad method name raises ValueError. No GPU, no network — it’s the floor under everything above.

Try it yourself:

  1. Run the second alternative (if you did ORPO, do KTO — or vice versa) and add its row to the eval.py comparison table. Now you can read DPO vs ORPO vs KTO on the same held-out task.
  2. Run SimPO via train_preference("simpo", ...) and vary the margin (simpo_gamma). Watch its effect on the average length of the generated tool calls — SimPO is length-normalized, so this is where its character shows.

In production

What changes once this is real, not a lab.

The choice is made on data, not on hype. The signal you actually collect in production is usually binary — accept/reject logs, 👍/👎 from users, the success or failure of a verified tool call. That’s the natural home of KTO, and manufacturing artificial pairs out of that binary signal wastes information and can add noise. ORPO earns its keep when you want one run from the base: no separate SFT pass, no resident reference model, which means less VRAM and a shorter pipeline — genuinely useful on a tight GPU budget. DPO stays the default when you have pairs, a prior SFT, and no special length or memory pressure.

The hidden cost is the experimental status. In trl 1.6.0 (June 2026), ORPOTrainer and CPOTrainer (SimPO) live in trl.experimental, and even KTOTrainer is flagged as migrating — their signatures can move between TRL releases without a long deprecation. So in production you pin trl hard (trl==1.6.0), keep a fast-failing test that breaks the instant an import path changes, and document the exact version in the model card. The opinionated default, stated plainly: I default to DPO because it’s the only fully-stable trainer; I move to KTO only when my feedback is truly binary; and I treat ORPO/SimPO as experiments I pin and re-verify on every upgrade.

One line of foreshadowing: the next step leaves preferences behind entirely. Module 10 stops asking which answer a human prefers and starts computing the reward — GRPO with verifiable rewards (RLVR), where a valid tool call simply scores 1.0.

Concept check

Why this matters. This module gives you the one skill that turns “I know four preference methods” into “I know which one to run”: reading the three axes — does it need a reference model, is the data pairwise or binary, is it single-stage — and mapping your situation onto a decision tree. It also inoculates you against two traps: assuming newer means better (it doesn’t — you choose on data and constraints), and assuming these trainers are as solid as DPO (they’re not — ORPO/SimPO are experimental, KTO is migrating). With this, you’ve closed the preference-tuning area (T8, Modules 8 + 9). Module 10 moves to verifiable RL. The questions below stay inside what this module taught — they’re all about choosing a method from the data you have.

  1. Your only feedback is a thumbs-up or thumbs-down on each individual model response — never matched pairs. Which method fits the data with no transformation, and why?

    • A. DPO — it’s the default, just pair up a random good and bad.
    • B. SimPO — it’s reference-free, so it doesn’t need pairs.
    • C. KTO — it consumes binary good/bad labels on single completions directly.
    • D. ORPO — it’s single-stage, so it handles any data.

  2. You want to align from the base model in a single run — no separate SFT pass, and no reference model resident in memory. Which method, and why?

    • A. KTO — it drops the reference model.
    • B. ORPO — its monolithic loss folds SFT in and needs no reference model.
    • C. SimPO — it’s reference-free, which means single-stage.
    • D. DPO — it’s the most efficient default.

  3. You already have an SFT checkpoint, clean (chosen, rejected) pairs, and you want the safest, best-understood, production-ready method. Which do you reach for?

    • A. ORPO — it’s newer than DPO, so it’s an upgrade.
    • B. KTO — it’s the most flexible.
    • C. DPO — it’s the only fully-stable trainer and exactly fits pairs + prior SFT.
    • D. SimPO — it controls length, which is always better.

  4. A teammate writes from trl import ORPOTrainer, then after a TRL upgrade hits an ImportError, and separately sees a warning that KTOTrainer is “migrating.” What’s going on, and what’s the fix?

    • A. TRL is broken; downgrade to an old release and never upgrade.
    • B. ORPOTrainer lives in trl.experimental.orpo, not the top level; KTO is importable from the top level but flagged as migrating. Pin trl and import from the verified path.
    • C. All preference trainers are stable; it’s a typo.
    • D. ORPO was removed from TRL; switch to PPO.

  5. You want reference-free training with length normalization on your (chosen, rejected) pairs. Which TRL object do you construct?

    • A. SimPOTrainer with length_norm=True.
    • B. DPOConfig(reference_free=True).
    • C. CPOConfig(loss_type="simpo", ...) with CPOTrainer — there is no SimPOTrainer.
    • D. KTOConfig(loss_type="simpo").

Answers.

  1. C. KTO is the one method that eats binary good/bad labels on single completions, with no pairing required. DPO/SimPO need pairs; pairing up a random good and bad (option A) is exactly the signal-wasting anti-pattern KTO exists to avoid. ORPO needs pairs too, and “single-stage” has nothing to do with handling unpaired data.

  2. B. ORPO’s monolithic loss combines the SFT objective with an odds-ratio preference penalty, so it runs in one stage from the base and needs no reference model — both constraints satisfied. KTO keeps a reference model (A is wrong). SimPO is reference-free but is not single-stage and doesn’t fold in SFT (C conflates the two). DPO needs a prior SFT and a reference model (D).

  3. C. DPO is the only fully-stable trainer here, it’s the best understood, and it’s designed for exactly this input: pairs plus a prior SFT. “Newer” (A) is not “better” and ORPO is experimental. KTO (B) wants binary data. SimPO (D) is for length control, not a general default, and is experimental.

  4. B. As of trl 1.6.0, ORPOTrainer is in trl.experimental.orpo — the top-level import fails. KTOTrainer can be imported from the top level but TRL warns it’s still migrating (“API changes may occur”). The fix is to import from the verified experimental path and pin the TRL version, with a test that fails loudly if the path moves — not to abandon upgrades (A) or switch to PPO (D), which is itself legacy/experimental.

  5. C. In TRL there is no SimPOTrainer. SimPO is a loss variant of CPO: CPOConfig(loss_type="simpo", ...) driven by CPOTrainer, from trl.experimental.cpo. DPOConfig has no reference_free field (B), and loss_type="simpo" belongs to CPO, not KTO (D).

Common pitfalls.

  • “Newer = better.” None of ORPO, KTO, or SimPO beats DPO universally — the paper benchmarks are their setups, not your task. Choose on the data format (pairs vs binary) and your constraints (reference model? single stage?), then measure on your own eval. DPO stays the safe default.
  • Treating the family as stable. As of trl 1.6.0 (June 2026), only DPOTrainer/GRPOTrainer are bedrock. ORPOTrainer and CPOTrainer (SimPO) are in trl.experimental; KTOTrainer imports from the top level but TRL flags it as migrating. Pin the version, document it in the model card, and date any version-dependent claim “as of trl 1.6.0, June 2026.”
  • Feeding pairs to KTO, or inventing SimPOTrainer. KTO wants binary {prompt, completion, label} — derive it from pairs if you must (make_kto_dataset), but know you’d normally collect it directly. SimPO is CPOConfig(loss_type="simpo"), never a dedicated trainer.

Key takeaways

  • The DPO family is chosen on three axes — reference model? data format (pairwise vs binary)? single-stage? — never on the paper’s publication date.
  • ORPO is single-stage: one monolithic loss (SFT + an odds-ratio penalty), no separate SFT pass and no reference model — but it still eats pairs.
  • KTO trains on binary good/bad labels (and keeps a reference model) — the right fit the moment your real feedback is genuinely binary, instead of faking pairs out of it.
  • SimPO is reference-free and length-normalized; in TRL it’s CPOConfig(loss_type="simpo") driven by CPOTrainer, not a SimPOTrainer.
  • The tree: pairs → DPO/SimPO; binary → KTO; one stage with no reference model → ORPO. When in doubt, DPO.
  • DPO stays the stable default. As of trl 1.6.0 (June 2026), ORPOTrainer/CPOTrainer are in trl.experimental and KTOTrainer is flagged as migrating — pin trl hard and re-verify on every upgrade.
  • Always re-evaluate on your task (Module 7). “Newer method” never guarantees “better” — on a small run the differences are small and noisy; only the eval tells you.

What’s next

You’ve shaped Anvil’s preferences every way the DPO family allows — pairs, binary labels, with and without a reference model, in one stage or two. Module 10 stops asking which answer a human prefers and starts computing the reward — GRPO with verifiable rewards (RLVR), where a valid tool call simply scores 1.0 and the gradient flows from a deterministic checker, not a label. That’s where alignment turns into reasoning.

Want the full fine-tuning concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · ← Module 8: Align It: Preference Tuning with DPO · Module 10: Teach It to Reason: GRPO and Verifiable Rewards → · Lab code for this module

References