aifromzero.dev

Teach It to Reason: GRPO and Verifiable Rewards (Fine-Tuning in Production, Module 10)

Module 10 of 15 16 min read Lab code ↗

This is Module 10 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Start at Module 1 or browse the full syllabus.

You handcrafted the preferences. Now let the model grade itself.

In Modules 8 and 9 you aligned Anvil with DPO and its family. It learned to prefer the right tool call — but only because you handed it the answer twice over. For every prompt you built a (prompt, chosen, rejected) triple: the gold call as chosen, a deliberately degraded call as rejected. That works, and it’s cheap compared to PPO. But it has a ceiling. Anvil never learns anything you didn’t already encode into a pair. Your preference data is the teacher, and a teacher who only knows two examples per question can only take you so far.

Here’s the thing that should bug you: for tool-calling, you don’t need a teacher at all. You can check whether an output is correct, mechanically, every time. Does the JSON parse? Is the tool name the one the request wanted? Do the arguments match the schema? That’s not a preference — it’s a verdict. So why are you spoon-feeding pairs when you could let Anvil generate a dozen attempts, score each one with a checker you already wrote, and push it toward the ones that pass?

That is exactly what GRPO does with verifiable rewards — the recipe that made DeepSeek-R1 famous. No reward model to train, no critic network, no preference pairs. Just a checker and a group of completions. One warning before you start the engine: this is the heaviest lab in the course. GRPO generates during training, so the GPU bill is real. We’ll size it honestly — a free T4 if it fits, a rented A100 if it doesn’t, with the cost written down.

In this module

You’ll learn how to:

  1. Explain what GRPO (Group Relative Policy Optimization) is by subtracting it from PPO: the same policy-gradient idea, minus the critic — each completion’s advantage is its reward minus the group mean (paper 2402.03300, DeepSeekMath). Why dropping the critic roughly halves the memory of an online RL step.
  2. Define a verifiable reward and RLVR (Reinforcement Learning with Verifiable Rewards, Tülu 3, 2411.15124): the reward is a deterministic checker, not a learned model — and why that needs a verifiable domain (math, code, structured output), with tool-calling as the ideal case.
  3. Build Anvil’s GRPO loop: wire the frozen verifiable reward_fn into a GRPOTrainer (TRL, stable), run it on the prompt-only GRPO dataset (make_grpo_dataset("train", ...) renders the train rows into {prompt, gold}), and watch the reward curve climb.
  4. Reason about cost: rollouts dominate (you generate a group of completions per prompt, every step) — use vLLM rollouts and Unsloth’s “80% less VRAM for GRPO” to fit a free T4, or fall back to a rented A100 with the cost recorded.
  5. Locate GRPO honestly: it’s the post-DeepSeek-R1 (2501.12948) reasoning recipe, not a universal trick — no verifiable domain, no GRPO — and PPO is now historical (trl.experimental.ppo, never the default).

You’ll build: Anvil now learns from a verifier, not a label. train_grpo.py wires the frozen reward_fn — 1.0 for a parseable tool call with the right name and valid args, partial credit for valid JSON, 0 otherwise — into a GRPOTrainer. Every step samples a group of completions per prompt, scores each one deterministically, and pushes Anvil toward the higher-reward completions relative to the group mean. No critic, no learned reward model. You’ll watch the reward curve climb, re-run the Module-7 eval to prove the gain, and see your lab.md print the real GPU cost.

Theory areas covered:

  • T9 — RL & reasoning: GRPO, RLVR, verifiable rewards — 9% of the course. Module 10 owns T9 outright; it’s reinforced once more at the capstone (Module 15).

This module builds on T8 (preferences, Modules 8–9, already done) — DPO is the contrast — and reuses T2 (the chat template / tools=, Module 2) to format prompts and T7 (task-grounded eval, Module 7) twice: once as the innards of the reward, once to prove the gain.

Prerequisites: Modules 1–9 — the stack, config.py/get_model_and_tokenizer (M3/M5), the tool-calling dataset and chat template (M2), the base-vs-fine-tuned eval (M7), and DPO alignment (M8). For the lab: a free Colab/Kaggle T4 (16 GB) as the first attempt (Unsloth GRPO), with a 💸 rented-GPU path (A100) documented if the T4 falls short (decision F5). HF_TOKEN set in your environment (M1).

Where you are

Anvil’s training stages, with this module marked:

  • ✅ M1–M2 — baseline + data (chat template, tools=).
  • ✅ M3–M6 — SFT, LoRA, QLoRA (fits a free 16 GB T4), Unsloth/Axolotl.
  • ✅ M7 — task-grounded eval (tool_call_valid_json, function_name_accuracy, argument_match).
  • ✅ M8–M9 — preference alignment (DPO default; ORPO/KTO/SimPO variants).
  • 👉 M10 — GRPO + verifiable rewards. Anvil learns from a deterministic verifier of its own sampled completions — no labels, no reward model.
  • ⬜ M11–M15 — merge (mergekit), quantize (GGUF/AWQ/GPTQ), serve (vLLM), publish (the Hub), capstone.

The one-line before/after for Anvil: it goes from “prefers the right tool call from hand-built preference pairs (DPO)” to “learns from a deterministic verifier of its own sampled completions (GRPO + RLVR).” The reward_fn you freeze here is replayed verbatim at the capstone (M15), and GRPO is the last training stage before the tail of the pipeline — merge → quantize → serve → publish.

From PPO to GRPO: drop the critic, score against the group

To understand GRPO, start from what it replaces. Classic RLHF — the alignment recipe everyone learned in 2023 — is PPO (Proximal Policy Optimization): a policy-gradient method that nudges the model toward higher-reward outputs. To do that stably, PPO needs an advantage — how much better than expected a given output was — and to compute that it carries a second network, a value model (also called the critic), trained alongside the policy to predict the expected reward (the baseline). So a PPO run holds two full models in memory — the policy you’re training and a critic of roughly the same size — and runs an online RL loop that is notoriously fiddly to keep stable.

That cost is exactly why DPO (Module 8) displaced PPO for preference alignment: DPO folds the reward into a closed-form loss over (prompt, chosen, rejected) pairs, so there’s no reward model and no online loop. But DPO only works on pre-labelled pairs. It never generates a candidate and grades it; it just leans toward the chosen you supplied. If you want the model to try things and learn from whether they were correct, you need an online method again — without paying PPO’s critic tax.

GRPO = Group Relative Policy Optimization (paper 2402.03300, DeepSeekMath) is that method. The central trick, at the “why” level: delete the critic. Instead of a value network estimating a per-token baseline, for each prompt you sample a group of G completions, score every one with the reward function, and use the group’s own mean as the baseline. The advantage of completion i becomes its reward normalized against its group:

advantage_i = (reward_i − mean(rewards)) / std(rewards)

A completion that beats its group’s average is reinforced; one that’s worse is pushed down. The baseline is no longer a learned network you have to train and keep calibrated — it’s just arithmetic over the rewards you already computed.

Why this matters concretely. Removing the critic removes an entire model from VRAM — the value network is about the size of the policy — so a GRPO step roughly halves the memory of an online RL step versus PPO (this is the practical contribution the DeepSeekMath paper sells; the exact savings depend on the implementation, as of trl 1.6.0, June 2026). And the group baseline is often more stable than a poorly-calibrated critic chasing sparse rewards: when most completions score zero, a value network struggles, but a group mean is still a sensible reference.

Here is the loop. Note the block that is absent — there is no value model anywhere in it:

flowchart TD
    P["prompt"] --> POL["policy (Anvil)"]
    POL -->|sample G completions| C["c1, c2, ... cG"]
    C --> R["reward_fn (verifiable)"]
    R --> RV["r1, r2, ... rG"]
    RV --> A["advantage_i = (r_i − mean(r)) / std(r)<br/>group baseline — NO critic"]
    A --> U["policy-gradient update"]
    U -.-> POL
    NOTE["PPO would add a value model here.<br/>GRPO does not."]:::ghost -.-> A
    classDef ghost fill:#fff,stroke:#bbb,stroke-dasharray:5 5,color:#888;

The object that runs it: GRPOTrainer

In TRL, GRPO is a top-level, stable trainer — not experimental. You import GRPOTrainer and GRPOConfig straight from trl:

from trl import GRPOTrainer, GRPOConfig

The config knobs that define a GRPO run (verified against GRPOConfig, trl 1.6.0, June 2026): num_generations is G, the number of completions sampled per prompt; max_completion_length caps how long each rollout runs; use_vllm toggles vLLM-accelerated rollouts (more on that below); and the usual learning_rate, beta, temperature. Here is Anvil’s config factory — every choice lives in config.py, never inline in a training script:

def grpo_args(output_dir: str = "outputs/anvil-grpo", **overrides):
    """GRPO is PPO without a value model: sample a *group* of num_generations
    completions per prompt, reward each, baseline = the group mean."""
    from trl import GRPOConfig
    defaults = dict(
        output_dir=output_dir,
        num_generations=4,            # G: completions sampled per prompt
        max_completion_length=256,
        per_device_train_batch_size=8,
        gradient_accumulation_steps=2,
        learning_rate=1e-6,
        use_vllm=False,               # off for CPU/CI; on for a real GPU run
    )
    defaults.update(overrides)
    return GRPOConfig(**defaults)

One constraint that trips people up: the effective batch (per_device_train_batch_size × gradient_accumulation_steps × world_size) must be divisible by num_generations, because TRL needs to fit whole groups into each optimization step. If it isn’t, you’ll get a config error before training starts.

The trainer itself takes the model, the reward function, the config, and a dataset of prompts — not preference pairs:

from trl import GRPOTrainer

trainer = GRPOTrainer(
    model=model,                  # from get_model_and_tokenizer(tier)
    reward_funcs=[reward_fn],     # TRL 1.6.0: a LIST of reward functions
    args=args,
    train_dataset=train_dataset,  # PROMPTS, not (chosen, rejected) pairs
    processing_class=tokenizer,   # TRL v1 idiom — never tokenizer=
    peft_config=peft_config,      # LoRA/QLoRA adapter for the policy
)
trainer.train()

Two idioms to internalize, because the internet is full of the stale versions. The reward is passed as reward_funcs=[reward_fn] — a list (TRL accepts several reward functions and can weight them via reward_weights). And the tokenizer goes in as processing_class=tokenizer, never tokenizer= — that’s the TRL v1 idiom, and tokenizer= will raise. As of trl 1.6.0, June 2026.

⚠️ Freshness note on “TRL v1.” The brief for this course originally pinned trl==0.29.1 and banned trl==1.0, on the theory that “TRL v1” was just branding. That’s now obsolete: the 1.x line shipped to PyPI and the current stable is trl==1.6.0. The v1 idioms — processing_class, max_length, stable GRPOTrainer, PPO living in trl.experimental.ppo — are unchanged and correct. Pin the exact version; don’t recite “never 1.0” from an old tutorial.

PPO vs GRPO at a glance

ComponentPPOGRPO
Policy model✅✅
Value model (critic)✅ — a second full network❌ — none
Baseline for the advantagethe critic’s predictionthe group mean of G completions
Reward signala learned reward model (RLHF)any reward function — here, a verifiable checker
Memory (order of magnitude)≈ 2× (policy + critic) + rollouts≈ policy alone + rollouts
Status in TRL (1.6.0, June 2026)trl.experimental.ppo — legacy 🧪stable, top-level from trl import GRPOTrainer

The row that matters for your bill is memory. The row that matters for your code is status: in 2026, GRPO is the stable, default online-RL path. PPO is historical — it lives in trl.experimental.ppo and emits a warning on import. Any tutorial that does from trl import PPOTrainer and calls PPO “the” RL method is teaching you 2023.

Verifiable rewards (RLVR): the reward is a checker, not a model

We’ve talked about how GRPO uses a reward. Now the conceptual pivot of the module: where the reward comes from.

In classic RLHF, the reward comes from a learned reward model — a network trained on human preference data to predict “good/bad.” That network is an approximation, it carries the biases of its training data, and the policy can learn to reward-hack it: find outputs the reward model scores highly but a human would not. Training, hosting, and auditing that reward model is a project in itself.

RLVR — Reinforcement Learning with Verifiable Rewards (the term popularized by Tülu 3, paper 2411.15124) flips this. The reward is a deterministic checker — a function that computes whether the output is correct (1 if checkably correct, less otherwise). No network, no learned approximation, no reward-model bias to police. You can read the reward; it’s code.

What makes a reward “verifiable”

The requirement is sharp: you need a programmable, deterministic oracle of correctness for the domain. The canonical verifiable domains:

  • Math — is the final answer equal to the known solution?
  • Code — do the unit tests pass?
  • Structured output / tool-calling — does the JSON parse, is the tool name right, do the arguments satisfy the schema?

And the canonical non-verifiable tasks: “write a touching poem,” “summarize this nicely.” There’s no deterministic oracle for “touching” or “nicely” — to score those you’d need an LLM-judge or a learned reward model, which is not RLVR. This boundary is the whole point, and we’ll hammer it again at the end of the module.

Why tool-calling is the ideal RLVR case

This is Anvil’s bet. Its task is measurable by construction: a tool call is right or wrong, and you can tell which with a parser — exactly the ingredients of a verifier. Better still, you already wrote that verifier in Module 7. eval.py has parse_tool_calls, function_name_accuracy, and argument_match. RLVR reuses that same notion of “a correct call” as a training signal, not just a metric. That’s the loop that closes the fil rouge: the task you can evaluate becomes the task you can train by RL.

Here is Anvil’s frozen reward — the contract you freeze in this module and replay at the capstone. It does not duplicate any parsing; it imports the shared logic from eval.py, so “correct” means the same thing when you train and when you grade:

def reward_fn(completions, gold=None, **kwargs) -> list[float]:
    """Verifiable reward for a tool call (RLVR).
    Passed to GRPOTrainer(reward_funcs=[reward_fn])."""
    n = len(completions)
    golds = gold if gold is not None else [None] * n
    rewards = []
    for completion, gold_json in zip(completions, golds):
        text = completion if isinstance(completion, str) else _last_content(completion)
        gold_calls = json.loads(gold_json) if gold_json else []
        calls = ev.parse_tool_calls(text)
        if not gold_calls:
            rewards.append(1.0 if not calls else 0.0)   # negative: reward NO call
        elif not calls:
            rewards.append(0.0)                          # expected a call, made none
        else:
            name = ev.function_name_accuracy(text, gold_calls)
            args = ev.argument_match(text, gold_calls)
            rewards.append(0.3 + 0.3 * name + 0.4 * args)
    return rewards

Three design choices to read off that code:

The signature. reward_fn(completions, gold=None, **kwargs) -> list[float]. completions is the list of sampled completions for the group; the function returns one float per completion. The gold keyword arrives as a per-example column of the GRPO dataset — TRL passes every dataset column other than prompt to the reward as a keyword whose value is the list of that column over the batch. That’s why gold is a list of JSON strings, one per completion, and why the data.py step JSON-encodes the expected calls (Arrow can’t store heterogeneous tool structs natively).

Staged reward shaping. The reward isn’t binary. A perfect call scores 1.0 (base 0.3 + name 0.3 + args 0.4). Credit is gated: arguments only score once the name matches (that’s how argument_match works — it looks for a predicted call whose name equals the gold name first), so a valid call with the wrong name earns exactly the base 0.3 — no name credit, and no argument credit either. Get the name right but the arguments partly wrong and you land in between: 0.6 (base + name) up to 1.0 as more arguments match. Prose with no call at all scores 0.0. This is deliberate, and it’s a “why” worth understanding: a binary 0/1 reward is sparse. Early in training, an untuned model almost never emits a perfect call, so a binary reward is nearly always 0 — there’s no gradient, nothing to climb. Staged credit gives a denser signal: the model is rewarded for producing valid JSON even before it has the call exactly right, so it has a foothold to improve from. (One of the “Try it yourself” extensions strips the partial credit so you can watch learning start more slowly.)

Negatives handled. When gold is empty (a “no relevant tool” example), the correct behavior is to make no call — so the reward is 1.0 for prose and 0.0 for a spurious call. This is the same abstention logic the Module-7 eval uses, reused as a reward.

The verifier loop, with its three branches:

flowchart TD
    P["prompt + tools"] --> G["Anvil generates a tool call"]
    G --> V{"verifier (deterministic)"}
    V -->|parses + right name + valid args| R1["reward 1.0"]
    V -->|valid call, right name, partial args| R2["reward 0.6 – 1.0 (partial)"]
    V -->|valid call, wrong name| R2b["reward 0.3 (base only)"]
    V -->|not a valid call| R3["reward 0.0"]
    R1 --> U["feeds the GRPO update"]
    R2 --> U
    R2b --> U
    R3 --> U
    NOTE["no learned reward model —<br/>just a checker you can read"]:::note -.-> V
    classDef note fill:#fff,stroke:#bbb,stroke-dasharray:5 5,color:#888;

Learned reward vs verifiable reward

Learned reward model (RLHF-PPO)Verifiable reward (RLVR)
Origin of the signala trained network that predicts “good/bad”a function you write that computes correctness
Deterministic?no — a noisy approximationyes — same input, same score
Needs human preference data?yes, to train the reward modelno
Reward-hacking riskhigh — the policy games the approximate modellower — but you can still game a lax checker (see In production)
Where it worksanywhere (but the signal is fuzzy)only in a verifiable domain (math, code, structured)
Example”is this answer helpful?”Anvil’s tool call: JSON valid + name + args

The cost of GRPO: rollouts dominate (a T4 with Unsloth, or a rented A100)

Here is the structural fact that makes this the heaviest lab in the course. At every step, GRPO generates G completions per prompt — the rollouts. Autoregressive generation is the expensive operation in any LLM workload; GRPO does it num_generations times per prompt, every single step, during training. SFT and DPO generate nothing while they train — they score fixed text. GRPO’s rollouts dominate the compute, the wall-clock, and the VRAM. Budget for that, not for the gradient step.

There are two levers to make GRPO practicable on a free GPU.

Lever 1 — vLLM rollouts. GRPOTrainer can hand the generation of rollouts to vLLM instead of plain transformers.generate, which is dramatically faster. You enable it with use_vllm=True in GRPOConfig (off by default, and off for CPU/CI). As of trl 1.6.0, June 2026. Note the scope: here vLLM is purely a rollout engine for training — it is not serving. vLLM as an inference server (with --enable-lora, latency/throughput benchmarks) is Module 13; don’t conflate the two.

Lever 2 — Unsloth’s “80% less VRAM for GRPO.” Unsloth (the optional accelerator from Module 6) advertises up to ~80% less VRAM for GRPO and ships a free Colab notebook that runs Qwen3-4B GRPO on a T4 (as of unsloth 2026.6.7, June 2026 — verify the claim and copy the FastLanguageModel signature from a current official notebook rather than reciting it; the signature moves between releases). This is the lifeboat for the free path. Unsloth stays optional — the canonical path is plain TRL/PEFT — but if you want to keep a 4B GRPO run inside a 16 GB T4, it’s the most credible route.

💸 The rented-GPU path (decision F5). Time for Bedrock-style honesty. If the free T4 isn’t enough — OOM even with QLoRA + Unsloth, or iteration so slow it’s useless — you rent a GPU and write down the cost. Ballpark rates as of June 2026 (re-check each provider’s page; they change weekly): RunPod $1.19/h for an A100 ($2.69/h for an H100 SXM), Lambda ~$1.99/h for an A100, Modal ~$2.50/h for an A100 billed by the second. (Together bills per token, not per hour, so it’s not on this table.) A small Anvil GRPO run — a 4B policy in QLoRA, a modest group, a few dozen steps — is tens of minutes on an A100 ≈ $0.50–$2 (confirm at your real run; present it as “on my run,” never invented).

The decision rule I follow: smoke-test the GRPO config on the free T4 (small G, 5–10 steps — just enough to confirm the loop runs and the reward is finite), then rent an A100 only for the run that counts. The dollar or two buys you hours and a clean reward curve.

Reducing the rollout bill (concrete levers, with the trade-off):

  • Lower num_generations. A smaller G means fewer rollouts per step, but a noisier group baseline (you’re estimating the mean from fewer samples). It’s a genuine trade-off. I start with the factory default num_generations=4 and short completions, and only push G higher once the reward curve is clearly moving — more samples buy a better baseline, but every extra sample is another rollout to pay for.
  • Bound max_completion_length. Generating short costs less; for tool calls, you don’t need 256 tokens of completion.
  • Reuse QLoRA for the policy (tier="qlora", Module 5) so the base weights fit in 4-bit.
  • Turn on vLLM rollouts.

Where GRPO fits — and where it doesn’t

The DeepSeek-R1 wave

GRPO went from a math-paper footnote to the most-searched fine-tuning topic of the year because of one model: DeepSeek-R1 (paper 2501.12948; also published in Nature, 2025). DeepSeek-R1-Zero learned to reason — long chains of thought, self-correction — essentially through RL with verifiable rewards on math and code, with no reasoning-demonstration SFT at all; the released R1 then added a small cold-start SFT stage before the RL. The “why” is the striking part: in R1-Zero the reasoning wasn’t taught by examples, it was rewarded whenever the final answer was verifiably correct, and the chains of thought emerged. That’s the context behind every “grpo tutorial” search today.

Anvil is not a math model — on purpose

Don’t expect an R1-style leap. Anvil isn’t a pure reasoning model, and that’s a deliberate course design choice (decision F2: a math-reasoning fil rouge would over-index on GRPO and under-exercise SFT/DPO). For Anvil, GRPO refines the tool-calling behavior that SFT and DPO already installed — it pushes toward verifiably correct tool calls, not toward long reasoning traces. Set the expectation accordingly: the win is a measurable improvement on JSON validity / name accuracy / argument match when you re-run the Module-7 eval, not a dramatic capability jump.

The hard limit: you need a verifiable domain

GRPO + RLVR apply only where a deterministic verifier exists. No verifier — style, tone, subjective “usefulness” — no RLVR. There you fall back to DPO/preferences (M8/M9) or an LLM-judge (M7, with its biases). This is non-negotiable, and it’s the heart of the misconception below.

⚠️ Common misconception: “GRPO is a general fine-tuning upgrade — and I’ll need to train a reward model for it.”

Wrong, twice.

(1) GRPO + RLVR only give you a signal in a verifiable domain — math, code, structured/tool-calling — where a deterministic checker exists. On a subjective task (style, tone), there’s nothing to verify, so reach for DPO/preferences or an LLM-judge instead. GRPO is not a button you press on any task to make a model better.

(2) RLVR does not need a learned reward model — that’s the entire point. The reward is a function you write and can read (the reward_fn). Tool-calling is the ideal case precisely because its correctness is checkable. And don’t drift back to PPO + a reward model as “the” RL method: PPO is trl.experimental.ppo (legacy 🧪); GRPO is the stable 2026 path.

Variants, briefly

The GRPO family is evolving — you’ll see DAPO and Dr. GRPO mentioned as recent refinements (corrections to length and normalization biases in the group-relative advantage). Treat them as directions to be aware of, not recipes to implement; their papers aren’t re-verified here, so we won’t cite arXiv IDs or detail them. Anvil sticks to plain GRPO.

Build it: GRPO-tune Anvil against a verifiable reward

Goal: GRPO-tune Anvil with the frozen verifiable reward_fn on the prompt-only GRPO dataset (make_grpo_dataset("train", ...) renders the train rows into {prompt, gold}), watch the reward curve climb, prove the gain with the Module-7 eval, and record the real GPU cost.

What you’ll see: the mean reward rising over a few (dozen) steps, the peak VRAM and wall-clock printed at the end, and a base-vs-fine-tuned eval showing the GRPO variant holding or beating the DPO variant on JSON validity / name accuracy / argument match.

The full, tested code lives in finetune-prod-labs/module-10. Its smoke test runs offline — no GPU, no network — so you can verify the pipeline before you rent anything.

Step 1 — The GRPO dataset is just prompts. data.make_grpo_dataset renders each row into {prompt, gold} via the chat template (M2). No chosen/rejected — the reward is the supervision:

def make_grpo_dataset(split, tokenizer, *, limit=None) -> Dataset:
    rows = load_rows(split, limit=limit)
    return Dataset.from_list([
        {"prompt": render_prompt(r, tokenizer),
         "gold": json.dumps(r["tool_calls"] or [])}   # JSON so Arrow stays happy
        for r in rows
    ])

Step 2 — The reward reuses the M7 parser. The frozen reward_fn (shown above) imports parse_tool_calls, function_name_accuracy, and argument_match from eval.py. Don’t re-implement parsing — share it, so “correct” means one thing.

Step 3 — Wire and train. train_grpo pulls the policy from the frozen get_model_and_tokenizer(tier), builds the dataset and config, attaches a LoRA adapter, and runs:

def train_grpo(*, tier="full", use_lora=True, data_limit=None,
               output_dir="outputs/anvil-grpo", **arg_overrides):
    from trl import GRPOTrainer
    model, tokenizer = config.get_model_and_tokenizer(tier)
    train_dataset = data.make_grpo_dataset("train", tokenizer, limit=data_limit)
    args = config.grpo_args(output_dir=output_dir, **arg_overrides)
    peft_config = config.lora_config() if use_lora else None
    trainer = GRPOTrainer(
        model=model,
        reward_funcs=[reward_fn],     # TRL 1.6.0: a LIST
        args=args,
        train_dataset=train_dataset,
        processing_class=tokenizer,
        peft_config=peft_config,
    )
    trainer.train()
    trainer.save_model(output_dir)
    return trainer

For a real GPU run, call train_grpo(tier="qlora", use_lora=True) (the main() default) and pass use_vllm=True to speed up rollouts.

Step 4 — Read the reward curve. The mean reward is logged each step (trainer.state.log_history). It should climb as Anvil produces more parseable, correctly-named calls. Measure peak VRAM with torch.cuda.max_memory_allocated() and record the wall-clock.

Step 5 — 💸 Rent if the T4 won’t hold. If you OOM or iterate too slowly on the free T4 even with Unsloth, switch to a rented A100 (RunPod/Lambda/Modal) and record $/h × wall-clock, dated June 2026. Smoke-test the config free, rent for the real run.

Step 6 — Re-evaluate (M7). Call compare_base_vs_finetuned(base_model, tuned_model, tokenizer, rows) with the DPO variant as base_model (your baseline) and the GRPO variant as tuned_model — the returned {"base": ..., "finetuned": ...} table is the GRPO-vs-DPO delta. This is the number that goes in Anvil’s model card (M14).

Step 7 — The offline smoke test. The suite runs the real GRPOTrainer for 2 steps on the tiny in-repo model with use_vllm=False — no GPU, no network — plus a pure unit test of reward_fn:

def test_reward_orders_outputs_correctly():
    r = reward_fn([PERFECT, WRONG_NAME, PROSE], gold=[GOLD, GOLD, GOLD])
    assert r[0] == 1.0                # perfect call
    assert 0.0 < r[1] < 1.0           # valid call, wrong name -> partial
    assert r[2] == 0.0                # no call at all
    assert r[0] > r[1] > r[2]         # the gradient GRPO follows

Run it from the repo root:

uv run pytest module-10/tests/
# 4 passed

Try it yourself:

  1. Vary num_generations (4 → 16) and watch its effect on the reward curve and on time/VRAM — quantify the trade-off between a better group baseline and more rollouts to pay for.
  2. Harden reward_fn to binary (1.0/0.0, no partial credit) and see whether learning starts more slowly — the sparse-signal problem that staged reward shaping fixes.

What this lab does NOT do. No learned reward model — the reward is the verifiable reward_fn (that’s the point of RLVR). No PPO — GRPO only; PPO is named as history (trl.experimental.ppo), never imported. No mergekit (M11), no GGUF/AWQ/GPTQ or merge_and_unload (M12), no vLLM serving (M13 — here vLLM is a rollout engine only), no Hub publish (M14). No R1-style long reasoning chains — Anvil refines its tool-calling. The real run needs an NVIDIA CUDA GPU; the smoke test runs on CPU.

In production

Outside the lab, GRPO is the most expensive stage of the pipeline, so cost/time becomes a first-class concern (decision F6). You isolate the GRPO run on dedicated or rented GPU, size G and completion length on purpose (every sample is a rollout you pay for), enable vLLM for the rollouts, and measure (reward curve + peak VRAM + $/h × wall-clock).

The number-one production risk specific to RLVR is a lax verifier the model learns to exploit — reward hacking of the checker, not of a learned reward model. A classic failure: the model discovers that emitting any technically-valid JSON earns the partial 0.3, so it games the partial credit instead of producing the right call. The defenses: audit and tighten the reward_fn, and keep partial credit small relative to the 1.0 so the model can’t farm it. And pin TRL — the GRPO API and the way dataset columns are passed to the reward can shift between versions (date everything as of trl 1.6.0, June 2026).

My own rule, stated as an opinion: I smoke-test the GRPO config on a free T4 with G=4 and 5 steps, then rent an A100 for the real run — the $1–2 buys me hours and a clean reward curve.

The next big lever is the rare one that’s free and GPU-less: merging Anvil’s variants (Module 11).

Concept check

Why this matters. This module owns T9. The skills it anchors: GRPO is PPO without the critic (advantage = reward minus the group mean, normalized), which roughly halves the memory of an online RL step; RLVR means the reward is a deterministic verifier, not a learned reward model; GRPO + RLVR only work in a verifiable domain, and tool-calling is the ideal case because correctness is checkable; and rollouts dominate the cost. And the freshness fact: PPO is now historical (trl.experimental.ppo), GRPO is the stable path.

  1. A colleague says: “GRPO is just PPO with a better reward model.” What’s actually different, and what’s the consequence?

    • A. Nothing — GRPO is PPO with a tuned reward model.
    • B. GRPO removes the critic/value model and uses the group mean as the advantage baseline; consequence: roughly half the memory of a PPO step.
    • C. GRPO adds a second critic for stability.
    • D. GRPO replaces the policy with a reward model.

  2. Four tasks are proposed for RLVR. Which is the poorest fit, and why?

    • A. Solve an algebra equation (final answer checkable).
    • B. Make a code change that must pass unit tests.
    • C. Emit a tool call that conforms to a JSON schema.
    • D. Write a touching poem about the sea.

  3. Why does tool-calling suit RLVR without a learned reward model?

    • A. Because tool calls are short, so they’re cheap to score.
    • B. Because correctness (valid JSON + right name + matching args) is checkable by a function you already wrote in Module 7, reused as the training signal.
    • C. Because the chat template guarantees a valid call.
    • D. Because DPO already made it correct, so any reward works.

  4. Your GRPO run saturates the free T4 and takes hours. What’s the cause, and the right lever?

    • A. Cause: full precision. Lever: switch the policy back to fp32.
    • B. Cause: the rollouts — G completions generated per prompt every step dominate the cost. Levers: lower num_generations/max_completion_length, enable vLLM rollouts, use Unsloth, or rent an A100.
    • C. Cause: too few epochs. Lever: raise num_train_epochs.
    • D. Cause: the reward function. Lever: remove it.

  5. A tutorial does from trl import PPOTrainer and calls PPO the default RL method. What’s wrong, and what should you do?

    • A. Nothing — PPO is still the stable default.
    • B. PPO moved to trl.experimental.ppo (legacy 🧪); GRPOTrainer is the stable 2026 path — use GRPO and treat PPO as history.
    • C. PPO was renamed GRPOTrainer; just rename the import.
    • D. You should pip install trl==1.0 to get PPO back.

Answers.

  1. B. GRPO deletes the value network and replaces the learned baseline with the group’s own mean — advantage_i = (r_i − mean(r)) / std(r). Dropping a full second model roughly halves the memory of an online RL step. The “better reward model” framing is wrong on two counts: GRPO changes the baseline, not the reward model, and with RLVR there’s no learned reward model at all.
  2. D. The poem has no deterministic oracle of correctness — “touching” isn’t checkable by a function, so it’s a poor RLVR fit (you’d need an LLM-judge or a learned reward model). The other three each have a programmable checker: equality to a known answer, passing tests, schema conformance.
  3. B. Anvil’s correctness is computable — valid JSON, right tool name, matching arguments — and that exact logic already exists in eval.py (M7). RLVR reuses it as a reward, so no learned reward model is needed. A is irrelevant (cost ≠ verifiability); C is false (the template doesn’t guarantee a correct call); D misunderstands RLVR.
  4. B. GRPO generates a group of completions per prompt at every step; those rollouts are the dominant cost in compute, time, and VRAM. The levers are smaller G, shorter completions, vLLM rollouts, Unsloth’s “80% less VRAM,” or the rented-A100 path. “Full precision” (A) is a red herring — Anvil’s policy is already QLoRA 4-bit.
  5. B. In TRL 1.6.0, PPO is experimental/legacy (trl.experimental.ppo) and GRPOTrainer is stable and top-level. Use GRPO; cite PPO only as historical context. (And trl==1.0 is the wrong fix — the current pin is trl==1.6.0.)

Common pitfalls.

  • “GRPO works on any task.” No — GRPO + RLVR need a verifiable domain (math, code, structured). Without a deterministic verifier (style, tone, subjective usefulness), there’s nothing to reward; use DPO/preferences (M8/M9) or an LLM-judge (M7). This is the conceptual trap of the module.
  • “You have to train a reward model for GRPO.” No — RLVR uses a hand-written verifier, not a learned reward model. That’s the inverse of classic RLHF-PPO. The reward_fn is a readable Python function.
  • from trl import PPOTrainer as “the” stable RL method. PPO moved to trl.experimental.ppo (legacy 🧪); GRPOTrainer is stable. And “TRL v1” is now the major version on PyPI — pin trl==1.6.0, not trl==1.0 (and the old “never 1.0” ban is obsolete). As of trl 1.6.0, June 2026.

Key takeaways

  • GRPO = PPO without the critic. Each completion’s advantage is (reward − group mean) / group std, so there’s no value network — which roughly halves the memory of an online RL step.
  • RLVR means the reward is a deterministic verifier, not a learned reward model. You write a function that computes correctness; there’s no reward model to train, host, or audit.
  • GRPO + RLVR need a verifiable domain (math, code, structured output). Tool-calling is the ideal case because its correctness is checkable.
  • Anvil’s reward_fn (1.0 / partial / 0) reuses the Module-7 eval logic as a training signal — staged credit gives a denser gradient than a binary reward.
  • Rollouts dominate the cost — G completions generated per prompt, every step. Use vLLM rollouts + Unsloth’s “80% less VRAM for GRPO” to fit a free T4; otherwise rent an A100 and write down the cost (💸 F5).
  • GRPOTrainer is stable in TRL 1.6.0; PPO is trl.experimental.ppo (legacy) — never the default. Pass the reward as reward_funcs=[reward_fn], the tokenizer as processing_class=.
  • DeepSeek-R1 (2501.12948) popularized the recipe, but GRPO is not a universal upgrade — no verifiable domain, no GRPO.

What’s next

Anvil has been hammered by SFT, QLoRA, DPO, and now GRPO — every stage cost GPU time. Module 11 is the rare free, GPU-less boost: merge your Anvil variants with mergekit (SLERP/TIES/DARE) and re-evaluate the result. Sometimes the best model isn’t one you train — it’s two you combine.

Want the full fine-tuning concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · ← Module 9: The DPO Family · Module 11: Model Merging with mergekit → · Lab code for this module

References