aifromzero.dev

Is It Any Better? Task-Grounded Evaluation (Fine-Tuning in Production, Module 7)

Module 7 of 15 14 min read Lab code ↗

This is Module 7 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Start at Module 1 or browse the full syllabus.

The loss went down. So what?

You finished Module 6. Anvil trained clean — the loss curve slid down, the notebook printed training complete, and you watched a QLoRA run finish on a free T4 in under an hour. You feel good. You paste in a request with a couple of tool schemas, and Anvil answers… roughly. It emits something that looks like a tool call. Close enough, right?

“Close enough” is not a metric. And here’s the trap that catches almost everyone next: you reach for one of two seductive non-proofs to reassure yourself. The first is “the loss went down” — but training loss measures next-token prediction on the data the model just memorized, not whether it does the job on inputs it has never seen. The second is “let me run MMLU — oh nice, the score went up two points, so it’s better” — but MMLU measures trivia and reasoning, not whether Anvil emits a valid tool call with the right name and the right arguments. Neither number answers the only question that matters.

That question is blunt: on a held-out set Anvil has never seen, does it produce more valid tool calls, with the right function name and the right arguments, than the base model does? This module builds the machinery that answers it — a numeric, base-vs-fine-tuned verdict. It is the table that turns “a model I trained” into “a model I can prove beats its base on its task,” and it is the table that makes Anvil’s model card credible in Module 14. This is the module the competition skips, and it’s the one that pays for the whole course.

In this module

You’ll learn how to:

  1. Reject the two seductive non-proofs — “the loss went down” and “MMLU went up, so it’s better” — and explain why neither measures whether Anvil is better on its task.
  2. Build a task-grounded evaluation on a held-out split: tool_call_valid_json (does it parse as a tool call?), function_name_accuracy (right tool?), and argument_match (right arguments?).
  3. Produce the base-vs-fine-tuned comparison — run Qwen/Qwen3-4B-Base and your fine-tuned Anvil through the same eval — the table that goes straight into the model card (Module 14).
  4. Measure catastrophic forgetting with a small general-capability bench via lm-eval, and read the trade-off: a sharper specialist that may have lost a little general ground.
  5. Use an LLM-as-judge (optionally, via lighteval + LiteLLM) as a signal, not ground truth — and name its biases: position, verbosity, and self-preference.

You’ll build: Anvil gets a verdict. You write anvil/eval.py with frozen metric signatures — tool_call_valid_json, function_name_accuracy, argument_match, evaluate, and compare_base_vs_finetuned — run the base model and your fine-tuned Anvil through the same held-out eval, and print the base-vs-fine-tuned table that lands in the model card. You add a small anti-forgetting bench with lm-eval, and optionally an LLM-as-judge score via LiteLLM. From now on, you re-run this after every training stage.

Theory areas covered:

  • T7 — Evaluation (task-grounded, benchmarks, anti-forgetting, LLM-judge) — 11% of the course. Module 7 owns this area, and it’s tied for the heaviest in the syllabus (with T3 and T8). Everything here is re-run, not re-taught, after DPO (M8), ORPO/KTO/SimPO (M9), GRPO (M10), merging (M11), and quantization (M12), all the way to the capstone (M15).

Prerequisites: Modules 1–6 — the stack installed; the frozen get_model_and_tokenizer(tier) factory from M3/M5 (you load both the base and the fine-tuned Anvil through it); the held-out eval split and apply_chat_template from data.py (M2); and a fine-tuned Anvil from M5/M6 — that’s the object you’re evaluating. HF_TOKEN set (M1). Optional for the LLM-judge: a provider key in .env via LiteLLM — the lab runs without it (the judge is a bonus, never a blocker). This whole lab is inference, not training, so it runs on a free T4.

Where you are

You’ve been forging Anvil for six modules. Here’s the map — ✅ done, 👉 you’re here, ⬜ ahead:

  • ✅ M1–M6 — Anvil curated, supervise-fine-tuned, fit onto a free 16 GB GPU with QLoRA, and optionally accelerated with Unsloth or described as Axolotl YAML.
  • 👉 M7 — Is It Any Better? Task-Grounded Evaluation. Anvil gets a numeric verdict.
  • ⬜ M8 — Align It: Preference Tuning with DPO.
  • ⬜ M9–M15 — ORPO/KTO/SimPO, GRPO, merging, quantization, serving, publication, capstone.

Anvil’s state going in: a Qwen3-4B fine-tuned in QLoRA whose loss went down — but unproven. We have no number that says it beats Qwen/Qwen3-4B-Base on the task, and no number that says it didn’t forget general ability on the way. Anvil’s state coming out: proven better on its task, with a base-vs-fine-tuned table in hand.

And one structural fact to lock in now: the eval.py you freeze here is the permanent measuring instrument of the series. It is re-run after DPO (M8), ORPO/KTO/SimPO (M9), GRPO (M10), the merge (M11), quantization (M12 — where you check the shrunk model didn’t regress), and the capstone (M15). The downstream modules call this eval; they never re-teach it. That’s why its signatures are frozen.

The two non-proofs: why loss and MMLU don’t tell you it’s better

Start by naming the two traps, because you will feel both of them pull at you.

Trap one: “the loss went down.” During SFT (Module 3), the trainer minimizes cross-entropy on next-token prediction over your training demonstrations. A falling training loss tells you the model is fitting the demonstrations — memorizing and interpolating them — not that it succeeds at the task on new inputs. An eval loss on a held-out split is already better, because at least it’s measured on data the model didn’t train on. But it’s still a proxy: averaged cross-entropy can look fine while the model emits malformed tool calls, because one stray brace that breaks JSON parsing barely moves a token-averaged loss. Loss is a training signal. It is not a task verdict.

Trap two: “MMLU went up, so it’s better.” This is the big one. MMLU (Massive Multitask Language Understanding) and HellaSwag are general-capability benchmarks — multiple-choice trivia, commonsense, reasoning. They do not measure whether Anvil emits a valid tool call with the right name and the right arguments. A model can climb on MMLU and stay useless at tool-calling, or the reverse. Reaching for MMLU to prove your tool-caller improved is answering a question nobody asked.

So here’s the concept the whole module hangs on, stated as a definition:

Task-grounded evaluation asks one thing: does the model do its actual job better? For Anvil, that job is producing tool calls that are valid, with the right function name and the right arguments, on inputs it has never seen.

That last clause is non-negotiable. The inputs come from the held-out eval split — the 200-row anvil_eval.jsonl you froze in data.py (Module 2), never touched during training. We won’t re-teach splits here; that was Module 2’s job. But the reason held-out is non-negotiable bears repeating: if your eval data leaks into your training data, the table you publish is a lie. A model that memorized the answers scores beautifully and helps no one. An honest verdict needs inputs the model has genuinely never seen.

Lay the measurement types side by side, because the point of the table is that only the bottom three rows answer the title’s question:

WhatMeasures whatExampleAnswers “better on its task?”When to use it
Training lossNext-token prediction on the train setthe loss curve from M3❌ No — a fit proxyWhile training, to see it converge
Eval lossNext-token prediction on held-out—⚠️ Partial proxyA sanity check, not a verdict
MMLU / HellaSwagGeneral capability (trivia, reasoning)culture, commonsense❌ No for the task / ✅ for forgettingAnti-forgetting check only (below)
tool_call_valid_jsonDid it emit a parseable call?held-out tool-calls✅ YesThe task floor
function_name_accuracyDid it pick the right tool?held-out tool-calls✅ YesThe right action
argument_matchDid it pass the right arguments?held-out tool-calls✅ YesThe subtle errors

The bottom three are Anvil’s task metrics, and they are the only ones that answer “is it better at its job?”. Here’s the pipeline you’ll build, end to end:

flowchart TB
    split["held-out eval split<br/>(data.py, M2 — anvil_eval.jsonl)"]
    split --> base["Qwen3-4B-Base<br/>get_model_and_tokenizer('full')"]
    split --> tuned["Anvil fine-tuned<br/>get_model_and_tokenizer + adapter"]
    base --> gen1["generate (greedy, temp=0)"]
    tuned --> gen2["generate (greedy, temp=0)"]
    gen1 --> score["parse + score:<br/>tool_call_valid_json<br/>function_name_accuracy<br/>argument_match"]
    gen2 --> score
    score --> cmp["compare_base_vs_finetuned"]
    cmp --> card["base-vs-fine-tuned table<br/>→ model card (M14)"]
    side1["also: lm-eval anti-forgetting bench"] -.-> cmp
    side2["optional: LLM-as-judge (LiteLLM)"] -.-> cmp

⚠️ Common misconception: “MMLU went up (or the loss went down), so my fine-tune is better.”

False. Training loss measures next-token prediction on the train set; MMLU measures general capability. Neither measures whether Anvil does its task — valid tool calls, right name, right arguments — better than the base. And the judge corollary: “an LLM-as-judge ruled it better, so it’s true” is just as false — the judge has biases (position, verbosity, self-preference); it’s a signal, not a verdict. Measure the task, on a held-out split, base-vs-fine-tuned.

The task metrics: valid JSON, function-name accuracy, argument match

This is the executable heart of the module. anvil/eval.py freezes five signatures (the contract from the fil-rouge spec). Every downstream module reuses them as-is, so any future change is by addition, never by breaking the names. Here they are, exactly as the lab freezes them:

def tool_call_valid_json(pred: str) -> bool: ...
def function_name_accuracy(pred: str, gold: list[dict] | None) -> float: ...
def argument_match(pred: str, gold: list[dict] | None) -> float: ...
def evaluate(model, tokenizer, rows: list[dict], *, max_new_tokens: int = 128) -> dict: ...
def compare_base_vs_finetuned(base_model, tuned_model, tokenizer,
                              rows: list[dict], **kwargs) -> dict: ...

Before the metrics, there’s one shared piece of plumbing: a parser. Anvil’s chat template renders calls as <tool_call>\n{"name": ..., "arguments": {...}}\n</tool_call> (Module 2 / the Qwen3 template), so the parser pulls the JSON out of those tags, with a fallback to any balanced {...} object that carries a name (it also normalizes the OpenAI {"function": {...}} shape). This same parser is shared with the GRPO reward in Module 10 — so “a correct call” means exactly one thing whether you’re grading Anvil or training it:

def parse_tool_calls(text: str) -> list[dict]:
    """Extract {name, arguments} calls. Prefers <tool_call>{...}</tool_call>; falls back to any
    balanced JSON object with a name/function. Shared by eval and the GRPO reward (M10)."""
    calls = []
    for match in _TOOL_CALL_RE.finditer(text):
        try:
            calls.append(json.loads(match.group(1)))
        except json.JSONDecodeError:
            pass
    if not calls:
        calls = [obj for obj in _balanced_json_objects(text)
                 if isinstance(obj, dict) and ("name" in obj or "function" in obj)]
    return [c for c in (_normalize(c) for c in calls) if c and c.get("name")]

Now the three metrics, from the floor up. Each one catches a different class of failure.

tool_call_valid_json(pred) — the floor. Did the model emit at least one parseable tool call?

def tool_call_valid_json(pred: str) -> bool:
    """Did the model emit at least one parseable tool call?"""
    return len(parse_tool_calls(pred)) > 0

Why it’s the floor: a malformed tool call is unusable by the caller — your runtime can’t dispatch JSON it can’t parse. This is failure number one for a base model. Remember Module 1: Qwen/Qwen3-4B-Base, handed a tool schema, produced prose or broken brace-soup, not a structured call. This metric is where fine-tuning wins the most ground, and it’s the first thing the table will show.

function_name_accuracy(pred, gold) — the right tool. Among the available tools, did it pick the correct one?

def function_name_accuracy(pred: str, gold: list[dict] | None) -> float:
    """Fraction of gold call names that appear in the prediction. For a negative (gold falsy),
    success means making NO call."""
    pred_calls = parse_tool_calls(pred)
    if not gold:
        return 1.0 if not pred_calls else 0.0
    pred_names = [c["name"] for c in pred_calls]
    matched = sum(1 for g in gold if g["name"] in pred_names)
    return matched / len(gold)

Why it matters: valid JSON that calls the wrong function is syntactically perfect and semantically wrong. The score is the fraction of gold call names that show up in the prediction. Notice the if not gold branch: when the example is a negative (a held-out row where the right move is not to call any tool — the abstention cases you curated in M2), success is making no call. A tool-caller that fires on a request it should decline is a real production failure, and this metric grades it.

argument_match(pred, gold) — the right arguments. “Right tool, wrong arguments” is the subtle error neither of the first two catches.

def argument_match(pred: str, gold: list[dict] | None) -> float:
    """Per gold call, key-to-key argument match (name must match first). Negative => no-call wins."""
    pred_calls = parse_tool_calls(pred)
    if not gold:
        return 1.0 if not pred_calls else 0.0
    scores = []
    for g in gold:
        match = next((c for c in pred_calls if c["name"] == g["name"]), None)
        gargs = g.get("arguments", {}) or {}
        if match is None:
            scores.append(0.0)
        elif not gargs:
            scores.append(1.0)
        else:
            pargs = match.get("arguments", {}) or {}
            ok = sum(1 for k, v in gargs.items() if k in pargs and pargs[k] == v)
            scores.append(ok / len(gargs))
    return sum(scores) / len(scores)

Here’s a design decision worth being explicit about, because hiding it would make the metric dishonest. Argument matching has two granularities: exact match (the whole arguments object must be identical — strict, severe) versus key-to-key match (the fraction of keys whose values are correct — more informative, more forgiving). This implementation uses key-to-key by default, so get_weather(city="Paris", unit="C") against gold get_weather(city="Paris", unit="F") scores 0.5, not 0.0. That’s a choice. It’s documented in the code and called out here on purpose — a metric whose strictness you can’t see is a metric you can’t trust.

Now the loop that runs them over a split:

def evaluate(model, tokenizer, rows: list[dict], *, max_new_tokens: int = 128) -> dict:
    """Score a model on Anvil's task. Returns aggregate metrics (the numbers that go in the card)."""
    pos = [r for r in rows if r.get("tool_calls")]
    neg = [r for r in rows if not r.get("tool_calls")]
    vj = fn = arg = abst = 0.0
    for row in rows:
        pred = generate(model, tokenizer, data.render_prompt(row, tokenizer),
                        max_new_tokens=max_new_tokens)
        gold = row.get("tool_calls")
        fn += function_name_accuracy(pred, gold)
        arg += argument_match(pred, gold)
        if gold:
            vj += 1.0 if tool_call_valid_json(pred) else 0.0
        else:
            abst += 0.0 if tool_call_valid_json(pred) else 1.0
    n = len(rows)
    return {"valid_json": vj / len(pos) if pos else None, "fn_acc": fn / n,
            "arg_match": arg / n, "abstention": abst / len(neg) if neg else None, "n": n}

Two evaluate details earn their place. First, the prompt is built with data.render_prompt(row, tokenizer), which runs apply_chat_template(messages, tools=..., add_generation_prompt=True) — never a hand-rolled template string. That’s the M2 contract: the template knows where the <tools> block and the generation prompt go for this model, and rolling your own is the classic way to silently mis-score the base model. Second, the returned dict carries n, the held-out size. A score over 20 examples does not carry the weight of a score over 2,000; you print n for statistical honesty. (You’ll also notice an extra abstention field — the fraction of negatives where Anvil correctly declined — and valid_json is computed over positives only, since you can’t emit a valid call on a row where the right answer is no call.)

Finally, the deliverable — the comparison itself:

def compare_base_vs_finetuned(base_model, tuned_model, tokenizer, rows: list[dict],
                              **kwargs) -> dict:
    """The base-vs-fine-tuned table that ships in Anvil's model card (decision F10)."""
    return {"base": evaluate(base_model, tokenizer, rows, **kwargs),
            "finetuned": evaluate(tuned_model, tokenizer, rows, **kwargs)}

It runs evaluate on both models — base and fine-tuned — and returns both dicts. This exact dict is what you paste into Anvil’s model card in Module 14 — that’s what makes the card credible (decision F10: the portfolio artifact is the published model with its base-vs-fine-tuned eval).

The same-decoding rule. Here’s the pitfall that quietly invalidates most home-grown comparisons: base and fine-tuned must be evaluated under identical conditions — same chat template, same generation params, same held-out split, same parser. Change any of those between the two runs and your delta measures the difference in decoding, not the difference in models. The generate helper enforces this by using do_sample=False (greedy), and compare_base_vs_finetuned feeds the same rows and kwargs to both. My opinion, stated with a number: I run eval at temperature=0 (greedy) so the comparison is reproducible — sampling adds noise that hides the real delta, and a verdict you can’t reproduce isn’t a verdict.

Here’s the shape of the table you’ll print. The numbers below are illustrative — these are your numbers, from your run, and you should never hard-code someone else’s:

MetricQwen3-4B-BaseAnvil (fine-tuned)Delta
tool_call_valid_json0.4X0.9X+0.XX
function_name_accuracy0.3X0.8X+0.XX
argument_match0.2X0.7X+0.XX
n (held-out)200200—

The story is the one Module 1 promised and this module proves: the base model is weak on the floor metric (malformed output), and the fine-tuned Anvil jumps well above it. But those are the numbers you measure, not numbers I’ll invent for you. Measure them at temperature=0 on your run, print the n, and that’s the table.

Catastrophic forgetting: did Anvil lose general ability?

A sharper specialist can be a narrower one. Catastrophic forgetting is the phenomenon where, in specializing, a model degrades its general abilities — reasoning, world knowledge, instruction-following outside its task. It’s the cost of aggressive fine-tuning: you bought a better tool-caller, and you may have paid for it with some general ground.

Why care for Anvil? A tool-caller that forgot how to parse an ambiguous user request is brittle in production. You don’t want to assume Anvil kept its general ability — you want to measure it.

The instrument is lm-eval (the lm-evaluation-harness, lm-eval==0.4.12, June 2026 — the de-facto standard that powers the Open LLM Leaderboard). You run a small general-capability bench — a few subtasks like a slice of MMLU and/or HellaSwag — on both the base and the fine-tuned model, and you read the delta. Why a mini-bench and not the full MMLU? On a free T4, the full MMLU is slow and burns GPU hours; a bounded subsample gives you the anti-forgetting signal without the bill (consistent with decision F6: be honest about cost). You bound it with --limit:

# mini anti-forgetting bench: a few subtasks, --limit to stay cheap on a T4
# (as of lm-eval 0.4.12, June 2026 — re-check the CLI against the installed wheel)
lm_eval --model hf --model_args pretrained=Qwen/Qwen3-4B-Base \
        --tasks hellaswag --limit 100 --batch_size auto

Then the same run on the fine-tuned model (the adapter merged or loaded onto the base). And here is the one rule that ties this section back to the misconception box: this is the only place MMLU and HellaSwag legitimately appear in this module — as a forgetting detector, never as proof that Anvil is “better.” The moment you cite an MMLU bump as evidence your tool-caller improved, you’re back in the trap.

A note on the wheel: lm-eval’s CLI and its simple_evaluate Python API have shifted between releases, and task names get renamed. Verify the exact flags against the installed lm-eval==0.4.12 before you write them down — don’t recite from memory. (The lab’s offline smoke test uses a cheap in-process anti_forgetting_probe proxy instead, so CI never needs a GPU; the real check is this lm-eval run on a T4.)

flowchart LR
    base["Qwen3-4B-Base<br/>task: low ▢▢▢<br/>general: high █████"]
    base -->|"fine-tune (M3–M6)"| anvil["Anvil fine-tuned<br/>task: high █████<br/>general: high-ish ████▢"]
    anvil --> note["specialization:<br/>task ↑, general ↓ (a little)<br/>measure BOTH, decide on purpose"]

The trade-off, in plain terms. Fine-tuning can push the task metric up and the general score down at the same time. That is not automatically a failure. A tool-calling specialist is allowed to lose a few points of MMLU if its task accuracy soars — that’s what “specialist” means. But you make that call knowingly, with both numbers in front of you. My take: a few points of MMLU for a big jump in valid tool-calls? For Anvil, I take that trade — but I write both numbers in the card so nobody’s surprised.

One honest, dated caveat: the magnitude of forgetting depends on the model and the recipe. A light QLoRA run tends to forget less than an aggressive full fine-tune, because the base weights stay frozen and only the small adapter moves. Treat that as a tendency, not a constant — and measure it on your run.

LLM-as-judge: a useful signal, never the ground truth

The three metrics above are automatic and objective — JSON parses or it doesn’t; the name matches or it doesn’t; the arguments match key-to-key or they don’t. For tool-calling, that’s exactly what you want, and it’s why Anvil is such a clean fil-rouge: the task is measurable without opinion. But some qualities are fuzzy — conciseness, tone, the judgment call of answering directly versus calling a tool when both are plausible. For those you can ask an LLM-as-judge: a model (often a stronger one) that scores responses against a rubric.

The tool here is lighteval (lighteval==0.13.0, June 2026) — a multi-backend evaluation framework (vLLM / Transformers / SGLang / LiteLLM) with LLM-as-judge built in and support for custom tasks. Why mention it here and not lm-eval? lighteval carries the judge and custom tasks out of the box; lm-eval stays our anti-forgetting instrument. Two complementary tools, no vendor lock (decision F11). The judge goes through LiteLLM, so any OpenAI-compatible provider works via a key in .env — never hard-coded. And it’s optional: the lab runs and passes without a key (clean skip). Here’s the hook the lab ships:

def llm_judge_score(prompt: str, prediction: str, *, model: str | None = None) -> float:
    """Score a prediction 0-1 with an LLM-as-judge over any OpenAI-compatible endpoint (LiteLLM).
    Needs JUDGE_API_KEY/JUDGE_MODEL. Beware judge biases (position, verbosity, self-preference);
    use it to complement, never replace, the task metrics above."""
    import os
    import litellm
    judge = model or os.environ["JUDGE_MODEL"]
    rubric = ("You are grading a tool-calling assistant. Given the user request and the "
              "assistant's answer, reply with a single number 0.0-1.0 for how correct the "
              f"tool call is.\nRequest:\n{prompt}\n\nAnswer:\n{prediction}\n\nScore:")
    resp = litellm.completion(model=judge, messages=[{"role": "user", "content": rubric}],
                              api_key=os.environ.get("JUDGE_API_KEY"))
    ...

Now the heart of this section: the biases. A judge is a model, and models have systematic tilts. Name them, and have a parade ready for each:

BiasWhat it isEffect on the scoreParade
Position biasFavors the answer shown first (or last)The order, not the content, winsSwap A/B order across two passes and average
Verbosity biasPrefers longer answers even when they add nothingPadding beats precisionRubric that explicitly doesn’t reward length — a tool call should be concise
Self-preference biasPrefers outputs from its own family/styleThe judge rewards its cousinsJudge from a different family than the model under test
InconsistencyDifferent scores across runsNoise masquerades as signalFix the judge model + a low temperature; repeat

The golden rule — and this is the judge corollary to the module’s whole thesis: a judge score is a directional signal, not ground truth. For Anvil, the objective metrics (valid JSON / function name / arguments) win; the judge only fills the gaps you can’t parse. My stance: I trust argument_match before I trust any judge — the judge is for the things I can’t parse, and even then I read it as a hint, not a verdict.

One forward pointer on custom tasks: lighteval lets you encode Anvil’s task metric as a custom task so you can replay it in a standardized harness. That’s a teaser, not a sub-tutorial — our hand-written eval.py stays the primary instrument (it’s the frozen contract the rest of the course calls). And don’t confuse this with Module 10’s GRPO reward function: the reward grades a completion during RL training; these metrics measure a model after it. Same parser, different jobs.

Build it: give Anvil a verdict

Goal: write anvil/eval.py (the five frozen signatures), evaluate Qwen/Qwen3-4B-Base against your fine-tuned Anvil on the held-out split, print the base-vs-fine-tuned table, add a small lm-eval anti-forgetting bench, and optionally an LLM-as-judge score — all on a free T4.

What you’ll see: a printed table of tool_call_valid_json, function_name_accuracy, and argument_match for base vs fine-tuned with deltas, plus a one-line anti-forgetting delta from lm-eval. The full, tested code lives in finetune-prod-labs/module-07.

Step 1 — Set up. Copy the Module 6 tree forward, then uv sync the locked core stack. The eval extras (lighteval, lm-eval) are installed per runtime (they manage their own torch/trl pins), so on your T4 box: pip install lighteval==0.13.0 lm-eval==0.4.12. Confirm the M6 smoke tests still pass offline. Have HF_TOKEN in your environment; optionally set JUDGE_API_KEY/JUDGE_MODEL in .env for the judge (never hard-coded).

Step 2 — Read the frozen metrics. Open anvil/eval.py: tool_call_valid_json, function_name_accuracy, argument_match, evaluate, and compare_base_vs_finetuned, plus the shared parse_tool_calls. These signatures are frozen — M8–M12 and M15 reuse them verbatim.

Step 3 — Compare base vs fine-tuned. Load both models through the frozen factory and the held-out split through data.py, then run the comparison at greedy decoding:

from anvil import config, data, eval as ev
from transformers import AutoModelForCausalLM

base_model, tok = config.get_model_and_tokenizer("full")          # Qwen3-4B-Base
tuned_model = AutoModelForCausalLM.from_pretrained("outputs/anvil-sft/merged")  # your merged M5/M6 weights
# (or keep the adapter separate: from peft import PeftModel;
#  tuned_model = PeftModel.from_pretrained(base_model, "outputs/anvil-sft"))
rows = data.load_rows("eval")                                      # the held-out split (anvil_eval.jsonl)
table = ev.compare_base_vs_finetuned(base_model, tuned_model, tok, rows)
print(table["base"], table["finetuned"])

Step 4 — Anti-forgetting with lm-eval. Run the bounded bench (--tasks hellaswag --limit 100) on base and tuned, read the delta, and comment honestly: a small drop is expected — you accept it on purpose. (Verify the CLI against the installed 0.4.12 wheel first.)

Step 5 — (Optional) LLM-as-judge. If a key is set, score a few responses via llm_judge_score with an anti-verbosity rubric and a position swap; otherwise it skips cleanly. The judge is a signal; the objective metrics win.

Step 6 — Run the offline tests. The smoke test asserts the metric logic on hand-built predictions (a valid call, broken JSON, wrong name, partial args) and runs evaluate/compare_base_vs_finetuned on the tiny fixture — no GPU, no network:

uv run pytest module-07/tests/
# 6 passed

What this lab does NOT do, on purpose. No DPO/ORPO/KTO/SimPO and no GRPO (M8–M10) — you evaluate a pre-preference Anvil and announce the re-eval after those stages; you train nothing new. No merge_and_unload, no GGUF/AWQ/GPTQ quantization (M11/M12) — you’ll note the eval verifies the quantized model later, but export nothing here. No model card or push_to_hub (M14) — you produce the table; you don’t publish it. No latency/throughput eval (M13) — this measures quality, not speed.

Try it yourself:

  1. Harden argument_match into strict exact match (the whole arguments object must be identical) and compare to key-to-key. Watch how the verdict shifts — that’s why you document your metric’s strictness.
  2. Add an lm-eval subtask (e.g. a slice of MMLU) to the anti-forgetting bench and plot the task↑ / general↓ trade-off across both dimensions (the second diagram, with your numbers).

In production

Out of the lab, four things change.

Eval is a decision instrument, not an end-of-notebook check. In production you freeze the held-out split and the parser, versioned like code (decision F14), so deltas between model versions are comparable over time. A held-out set that drifts silently makes every comparison meaningless — you’d be measuring the data, not the model.

Eval drives the rest of the series. You re-run it after every stage: DPO (M8), ORPO/KTO/SimPO (M9), GRPO (M10), the merge (M11), and quantization (M12) — where you specifically check the shrunk model didn’t regress. It’s the only way to know whether a step helped or hurt.

Cost (decision F6). Task eval is light — inference over a few hundred held-out examples on a free T4. The lm-eval bench is bounded with --limit. An LLM-judge over an API has a real per-token cost — date it “as of June 2026,” and only judge what you can’t measure objectively, or you’re paying for nothing.

Statistical honesty. Print n, fix the decoding to greedy, and repeat if the held-out set is small. The stance I’ll defend: a model card without a held-out, base-vs-fine-tuned table is marketing, not evidence. The table is the whole point.

Forward pointer: in Module 8 you align Anvil with DPO — and the first thing you’ll do afterward is re-run this exact evaluate.

Concept check

Why this matters. This module gives you the one skill that turns “I trained a model” into “I can prove it’s better”: measuring the task, not the loss or MMLU. You learned why a falling loss and a rising MMLU prove nothing about a tool-caller, what the three task metrics catch (valid JSON → right name → right arguments), why base-vs-fine-tuned on a held-out split with the same decoding is the only honest verdict, what catastrophic forgetting is and how to measure it with a bounded lm-eval bench, and why an LLM-as-judge is a biased signal — never ground truth. You also lean on Module 2 (the held-out eval split and apply_chat_template, reused as-is) and Modules 3/5 (the fine-tuned Anvil you’re grading, loaded via the frozen factory — you don’t re-train it). And remember the frame: this eval.py is re-run after every stage through the capstone, and its table is what makes Anvil’s model card credible.

  1. A developer says: “I fine-tuned Anvil, ran MMLU, and the score went up two points — so my tool-caller is better.” What’s wrong, and what should they measure?

    • A. Nothing — the MMLU score is the proof.
    • B. They should run a bigger MMLU to be sure.
    • C. The loss also went down, so it’s already doubly proven.
    • D. MMLU measures general capability, not the tool-calling task; measure tool_call_valid_json / function_name_accuracy / argument_match on a held-out split, base-vs-fine-tuned.

  2. During SFT, Anvil’s training loss dropped sharply. Is that proof the task is done better?

    • A. Yes — a lower loss is a better model.
    • B. No — training loss measures next-token prediction on the train set; a model can have low loss and still emit malformed tool calls. Measure the task on held-out data.
    • C. Yes, as long as the loss is below 0.5.
    • D. Only if you also checked the eval loss is identical to the train loss.

  3. Your LLM-as-judge consistently scores the longer of two answers higher, even when the extra text adds nothing. What is this, and what do you do?

    • A. Position bias — swap the answer order and average.
    • B. Self-preference bias — use a judge from the same family.
    • C. Verbosity bias — write a rubric that explicitly doesn’t reward length (and remember a tool call should be concise); treat the judge as a signal, not a verdict.
    • D. The judge is simply correct — longer answers are more thorough.

  4. After fine-tuning, Anvil’s task metric jumped but its MMLU score dropped a few points. How do you read it, and what do you do?

    • A. A bug — fix it before shipping anything.
    • B. Ignore it; only MMLU matters.
    • C. Catastrophic forgetting — measure it with a bounded lm-eval bench, decide on purpose (a specialist may trade a little general ability), and document both numbers in the card.
    • D. Re-train Anvil directly on MMLU to recover the points.

  5. You evaluate the base model at temperature=0.7 and the fine-tuned model at temperature=0, on two different held-out sets. Why is the resulting table wrong, and how do you fix it?

    • A. It’s fine — the models are different anyway.
    • B. Different decoding and different held-out sets mean the delta measures decoding/data, not the model; use the same greedy decoding, the same held-out split, and the same parser for both.
    • C. Only the temperature matters; the held-out set can differ.
    • D. Raise both temperatures to 1.0 for fairness.

Answers.

  1. D. MMLU is a general-capability benchmark; it can rise while the tool-calling task is unchanged or worse. The only verdict is the task metrics on a held-out split, base-vs-fine-tuned. A bigger MMLU (B) measures more of the wrong thing; the loss (C) is the other non-proof.
  2. B. Training loss is a fit signal on the train set, not a task verdict on new inputs — and averaged cross-entropy barely notices a brace that breaks JSON parsing. Measure the task on held-out data. The thresholds in C and D are arbitrary and irrelevant.
  3. C. Verbosity bias. The parade is an explicit anti-length rubric (and for Anvil, concision is correct — a tool call shouldn’t ramble). Position bias (A) is about order; self-preference (B) is about model family; D is the exact mistake the module warns against — never take a judge’s score as truth.
  4. C. That’s catastrophic forgetting, and it’s expected from specialization. Measure it (bounded lm-eval), make the trade knowingly, and write both numbers down. It’s not a bug (A), not ignorable (B), and re-training on MMLU (D) would chase a benchmark you don’t care about.
  5. B. Two things differ between the runs — decoding and data — so the delta is contaminated; it reflects the comparison setup, not the model. Fix the decoding to greedy, use one held-out split, and one parser for both. C ignores the split problem; D adds more sampling noise.

Common pitfalls.

  • “MMLU/the loss went up/down, so it’s better.” The central conceptual trap (and the misconception box). Neither measures the task. Measure tool_call_valid_json / function_name_accuracy / argument_match on a held-out split, base-vs-fine-tuned.
  • Taking an LLM-as-judge at its word. It has biases — position, verbosity, self-preference. It’s a signal, not a verdict; for tool-calling, the objective metrics win.
  • An unfair comparison. Different decoding, held-out, or parser between base and tuned makes the delta meaningless. Fix the decoding (greedy), freeze the held-out split, and print n.
  • Freshness. Pin and date your eval tooling: lighteval==0.13.0 and lm-eval==0.4.12 (June 2026). The lm-eval CLI and task names shift between versions — verify against the installed wheel, never recite from memory. And never hard-code measured scores: they’re your numbers, from your run.

Key takeaways

  • “The loss went down” and “MMLU went up” do not prove a fine-tune is better — they measure fit and general capability, not the task. Measure the task.
  • Anvil’s three task metrics run from floor to fine grain: tool_call_valid_json (parseable?) → function_name_accuracy (right tool?) → argument_match (right arguments, key-to-key by default?).
  • Base-vs-fine-tuned on a held-out split, with the same greedy decoding and parser, is the only honest verdict — and that table goes straight into Anvil’s model card (Module 14).
  • Catastrophic forgetting is measured with a bounded lm-eval mini-bench (MMLU/HellaSwag) — that’s their only legitimate role here, as a forgetting detector, never as proof of “better.”
  • An LLM-as-judge is a biased signal — position, verbosity, self-preference — never ground truth. For tool-calling, the objective metrics win; the judge fills the fuzzy gaps.
  • evaluate returns n for statistical honesty, and compare_base_vs_finetuned is re-run after every stage (DPO, ORPO/KTO/SimPO, GRPO, merge, quantization, capstone) — it’s the permanent measuring instrument of the series.

What’s next

Anvil is proven now — you have the table. Module 8 makes it prefer the right tool call (and stay concise) with DPO — and the first thing you’ll do afterward is re-run this exact eval to see if it actually helped.

Want the full fine-tuning concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · ← Module 6: Go Faster: Unsloth and Config-as-Code · Module 8: Align It: Preference Tuning with DPO → · Lab code for this module

References