Your First Fine-Tune: SFT from Scratch (Fine-Tuning in Production, Module 3)

This is Module 3 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Start at Module 1 or browse the full syllabus.

The TypeError that 70% of SFT tutorials hand you

You finished Module 2 with a clean dataset. Tool-calling demonstrations, formatted with Qwen3’s chat template, split into train and eval. You have a base model that can talk but can’t call a tool. The fix is supervised fine-tuning, so you open the first SFT tutorial a search engine hands you and copy the snippet:

trainer = SFTTrainer(model, tokenizer=tok, dataset_text_field="text", max_seq_length=1024)

You run it. You get a TypeError. The tutorial was written in 2024, and since then TRL removed tokenizer= (it’s processing_class= now), moved the dataset fields onto SFTConfig, and renamed max_seq_length to max_length. Every one of those three lines is wrong as of June 2026, and roughly 70% of the SFT content on the internet still ships the dead signature.

This is the main course of fine-tuning. You take a model that knows language but not your task, and you teach it the task by example. This module forges Anvil v0 — the first trained version of our tool-calling specialist — and it does it with the idioms that actually run today, not the ones that 404 your training loop on line one.

In this module

You’ll learn how to:

1. Explain what supervised fine-tuning actually optimizes — the next-token loss on demonstration data — and why that changes a model’s behavior and format, not its facts.
2. Write a correct TRL v1 SFT run with SFTTrainer(model, args=SFTConfig(...), processing_class=tokenizer) — knowing exactly which arguments live on the Trainer and which are fields of the SFTConfig.
3. Choose the core hyperparameters with intent — learning rate, epochs, per-device batch size, gradient accumulation, max_length — and frame the choice between full fine-tuning and PEFT.
4. Read a training loss curve: tell a healthy descent from a flat loss, a diverging loss, and overfitting, and know what to change for each.
5. Build Anvil v0 — freeze the model/tokenizer factory, run SFT, save the checkpoint, and generate on the same tool-call prompt before and after to see the fix with your own eyes.

You’ll build: Anvil v0. A frozen anvil/config.py factory that loads Qwen/Qwen3-4B-Base from one place, an anvil/train_sft.py that runs SFTTrainer/SFTConfig the TRL v1 way, a saved checkpoint, a before/after generation that turns the base model’s broken tool-call into a valid one, and the loss curve that proves training happened.

Theory areas covered:

• T3 — Supervised fine-tuning (SFT) — 11% of the course. This is the heaviest construction area in the syllabus, co-owned with the Module 15 capstone. We cover all of it here: the loss, the SFTTrainer/SFTConfig API, hyperparameters, reading a loss curve, full FT vs PEFT, completion-only loss, and the model factory.
• Builds on T2 (data and chat templates) from Module 2 — we reuse anvil/data.py exactly as you froze it, and we lean on the chat template to explain the single most common SFT failure: a flat loss.

Prerequisites: Module 1 (setup, HF_TOKEN, the Qwen3-4B-Base baseline) and Module 2 (anvil/data.py, chat templates, the train/eval splits). For the real run you want a free T4 16 GB GPU (Colab or Kaggle); the offline smoke test runs with no GPU at all.

Where you are: forging Anvil v0

You’ve been here before, and you’re about to leave with a model. Here’s the map — ✅ done, 👉 you’re here, ⬜ ahead:

• ✅ M1 — When (Not) to Fine-Tune. Setup; the base model fails the tool-call task (the baseline).
• ✅ M2 — Data Is the Model. anvil/data.py: the tool-calling fixture, chat-templated.
• 👉 M3 — Your First Fine-Tune: SFT from Scratch. Anvil v0: config.py + train_sft.py, the loss curve.
• ⬜ M4 — LoRA: Train 1% of the Weights. Swappable adapters via one peft_config.
• ⬜ M5 — Make It Fit a Free GPU: QLoRA. 4-bit NF4 base; Anvil fits a T4.
• ⬜ M6 — Go Faster: Unsloth and Axolotl. The same run, 2× faster.
• ⬜ M7 — Is It Any Better? Task-Grounded Evaluation. JSON validity, fn-name accuracy, arg-match.
• ⬜ M8–M9 — Align It: DPO and the preference family.
• ⬜ M10 — Teach It to Reason: GRPO and verifiable rewards.
• ⬜ M11–M14 — Merge, quantize, serve, publish.
• ⬜ M15 — Capstone: Anvil, end to end.

You arrived with a dataset and a base model that can’t tool-call. You’ll leave with Anvil v0: a fine-tuned model that emits valid tool-calls, plus a frozen factory the rest of the course reuses. That factory — anvil/config.py and its get_model_and_tokenizer(tier) — is laid down here and reused by every training module after it: M4 (LoRA), M5 (QLoRA), M6 (Unsloth), M8 (DPO), M10 (GRPO), and the M15 capstone. It is the foundation every later run stands on, which is exactly why we freeze it now.

What SFT actually learns: next-token loss on demonstrations

Supervised fine-tuning (SFT) is the act of continuing to train a pretrained model on demonstrations — input→desired-output pairs — using the exact same objective the model was pretrained with: the next-token loss. That’s it. There is no new objective, no new kind of learning. SFT does not “understand” anything it didn’t before. It nudges the model’s token probabilities toward the format and behavior in your demonstrations.

The next-token loss is causal cross-entropy. For each position in a sequence, the model produces a probability distribution over the vocabulary for the next token; the loss is the negative log-probability it assigned to the actual next token (the “gold” token). Average that over every position, backpropagate, take an optimizer step. Pretraining does this over trillions of tokens of raw text. SFT does the identical computation over your few thousand demonstrations. Only the data changed. This is precisely why fine-tuning changes a model’s behavior and format but not its facts: you are reweighting which continuations are likely, not inserting a knowledge base. (If you need facts, that’s retrieval — the prompting → RAG → fine-tuning escalation ladder from Module 1. Confusing the two is the wrong tool for the job.)

Why “supervised” and where it sits in the pipeline

The historical anchor here is InstructGPT (Ouyang et al., arXiv 2203.02155). It’s the paper that defined the modern post-training recipe, and its first stage is supervised fine-tuning on human demonstrations — before any reward model, before any reinforcement learning. The order matters and it’s the mental model for this whole course: you first align the model to the task and format by demonstration (SFT, this module), and only then refine it by preferences (DPO, Module 8) and by verifiable reward (GRPO, Module 10). SFT is the foundation the later stages stand on; skip it and the preference stages have nothing coherent to prefer.

One note on terminology, because it trips people up: InstructGPT’s third stage was PPO, a reinforcement-learning algorithm. In 2026 PPO is not the default follow-up to SFT — it lives in trl.experimental.ppo, and the stable, recommended next steps are DPO and GRPO. We cite InstructGPT for the shape of the pipeline (SFT comes first), not as the alignment method to copy.

Which tokens does the loss train on?

A demonstration is a full conversation: the user request, the available tools, and the assistant’s gold answer. By default the next-token loss is computed over the whole sequence — prompt and completion. But for a task like tool-calling, the prompt (user + tool schemas) is given; the only thing you actually need the model to produce is the assistant’s tool-call. Training on the prompt tokens spends gradient signal teaching the model to echo tool schemas back — a skill you don’t need.

So we mask it. completion_only_loss=True (a field on SFTConfig) tells TRL to compute the loss only on the completion tokens — the assistant turn — and ignore the prompt. The model learns to answer, not to parrot the question. Our anvil/data.py produces prompt and completion string columns precisely so this masking has a clean boundary to work with (more on that below). As of trl 1.6.0, June 2026, completion_only_loss defaults to None (auto-detected from the dataset shape); Anvil sets it to True explicitly because we always want the prompt masked.

Here’s the whole loop in one picture:

flowchart TD
    A["Demonstrations<br/>{prompt, completion}<br/>chat-templated (M2)"] --> B[Tokenize]
    B --> C["Forward pass<br/>(base model)"]
    C --> D["Next-token loss<br/>vs gold tokens<br/>(causal cross-entropy)"]
    D --> E["Backward pass +<br/>optimizer step"]
    E -->|loop over batches / epochs| C
    E --> F["Fine-tuned model<br/>(Anvil v0)"]
    D -.->|completion_only_loss=True| G["loss masked to the<br/>assistant tokens only"]
    style F fill:#d4edda
    style G fill:#fff3cd

The annotation on the left is the one-line summary of the whole module: same objective as pretraining — only the data changed. The dashed branch is completion_only_loss masking the prompt away.

The TRL v1 way: SFTTrainer + SFTConfig

This is the freshness heart of the module. Get the split between the Trainer and its Config right and your run works; get it wrong and you get the TypeError from the intro.

SFTTrainer — the high-level wrapper

SFTTrainer is TRL’s high-level wrapper around the transformers Trainer. It handles the SFT-specific work — tokenizing your dataset, optionally packing it, applying the chat template, masking the prompt — and then runs a standard training loop. As of trl 1.6.0, June 2026, it’s stable (top-level from trl import SFTTrainer); so are DPOTrainer, GRPOTrainer, and KTOTrainer. Only ORPO/CPO/BCO/PPO sit under trl.experimental.

The arguments you pass to the Trainer itself are the things, the data, and the tokenizer — not the configuration:

from trl import SFTTrainer

trainer = SFTTrainer(
    model=model,
    args=args,                    # an SFTConfig — everything configurable lives here
    train_dataset=train_dataset,
    processing_class=tokenizer,   # TRL v1: processing_class, never tokenizer=
    peft_config=peft_config,      # None for full SFT; a LoraConfig in Module 4
)

Three of those deserve a beat:

• processing_class=tokenizer is the argument that receives the tokenizer. Never tokenizer=. That keyword was removed in TRL v0.16; passing it today raises a TypeError. The rename happened because the same Trainer now accepts tokenizers, image processors, and feature extractors — a single “processing class” slot — so transformers 5.x uses processing_class too. Burn this in: the tokenizer goes in processing_class.
• peft_config is the slot where a LoraConfig goes to turn this into a LoRA run. It’s None here — Module 3 trains the full weights. We’re mentioning the slot so you recognize it in Module 4, not using it.
• formatting_func (not shown) is an optional hook that turns a raw example into text. We don’t need it: anvil/data.py already rendered each example through the chat template into prompt/completion columns, and SFTTrainer consumes those directly.

SFTConfig — where everything configurable lives

SFTConfig inherits from transformers’ TrainingArguments, and everything that configures training or dataset handling lives on it, passed in via args=. That includes the fields the dead tutorial tried to pass to the Trainer. Here are the ones that matter, with Anvil’s defaults (verified against the installed trl 1.6.0):

• output_dir — where the checkpoint is saved.
• max_length=1024 — the maximum sequence length. Not max_seq_length. That field was removed in v0.20 and raises a TypeError. Tool schemas plus a call fit comfortably in 1024 tokens; the default is also 1024.
• completion_only_loss=True — mask the prompt, train on the assistant turn (above).
• packing=False — whether to concatenate several short examples into one sequence for throughput. Packing speeds training but blurs example boundaries, which matters for tool-calling where each example is a self-contained call; we leave it off.
• The hyperparameters: learning_rate, num_train_epochs, per_device_train_batch_size, gradient_accumulation_steps, logging_steps, seed, and the scheduler (warmup_ratio, lr_scheduler_type).

Anvil bundles these defaults into one helper so they live in a single place:

def sft_args(output_dir: str = "outputs/anvil-sft", **overrides):
    """Build the SFTConfig for Anvil's SFT. TRL v1 idioms only: max_length (never
    max_seq_length); dataset/length/packing fields live on the *config*, not the Trainer."""
    from trl import SFTConfig
    defaults = dict(
        output_dir=output_dir,
        max_length=1024,
        completion_only_loss=True,
        num_train_epochs=1,
        per_device_train_batch_size=8,
        gradient_accumulation_steps=2,
        learning_rate=2e-4,
        warmup_ratio=0.03,
        lr_scheduler_type="cosine",
        logging_steps=10,
        save_strategy="epoch",
        seed=0,
        report_to=[],
    )
    defaults.update(_precision_kwargs())  # bf16 on Ampere+, fp16 on T4, neither on CPU
    defaults.update(overrides)
    return SFTConfig(**defaults)

One detail that bites everyone the first time: precision. We don’t hard-code bf16=True for the trainer. A free T4 has no bf16 tensor cores (bf16 is Ampere and up — A100/L4), and on a CPU a hard-coded bf16=True raises ValueError: Your setup doesn't support bf16. So _precision_kwargs() picks what the hardware actually supports — bf16 on Ampere+, fp16 on a T4, and neither on CPU — which is also why the offline tests run anywhere. One nuance worth naming: the model weights still load in bf16 (config.py loads the "full" tier with dtype=torch.bfloat16), while it’s the trainer’s compute precision — the autocast TRL uses for the forward/backward — that _precision_kwargs() auto-detects (bf16 on Ampere+, fp16 on a T4). Two different knobs: storage dtype vs compute precision.

Trainer argument vs SFTConfig field — the table to memorize

This table is the answer to the question that breaks copy-pasted tutorials. When in doubt, the rule is: the model, the data, and the tokenizer go on the Trainer; everything you’d tune goes on the Config.

Goes on the SFTTrainer (constructor arg)	Goes on the SFTConfig (via args=)
model	output_dir
processing_class=tokenizer (never tokenizer=)	max_length (never max_seq_length)
train_dataset / eval_dataset	dataset_text_field
peft_config (None here; LoraConfig in M4)	packing
formatting_func (optional)	completion_only_loss
	learning_rate, num_train_epochs
	per_device_train_batch_size, gradient_accumulation_steps
	logging_steps, seed, report_to, scheduler fields

⚠️ Common misconception: “I’ll just pass tokenizer=, dataset_text_field=, and max_seq_length= to SFTTrainer, like the tutorial shows.”

All three are wrong as of trl 1.6.0, June 2026. tokenizer= was removed in v0.16 — the tokenizer goes in processing_class=. dataset_text_field, packing, and completion_only_loss are fields of SFTConfig, not Trainer arguments — pass them via args=SFTConfig(...). And max_seq_length was renamed to max_length (removed in v0.20 → TypeError). This is the single most common fine-tuning bug in 2026, because roughly 70% of tutorials still show the pre-v1 signature. When you copy a snippet, check it against the installed version, not the blog post’s date.

A note on the data shape

The original brief for this module assumed a single pre-formatted text column (dataset_text_field="text"). Anvil’s frozen data.py does something cleaner: it renders each example into two string columns — prompt and completion — via the chat template. The reason is concrete and worth knowing: real tool schemas are heterogeneous, and Arrow (the storage layer behind 🤗 datasets) can’t infer one struct type across them. Two flat string columns are Arrow-safe and give completion_only_loss=True a clean boundary to mask on. SFTTrainer accepts prompt/completion datasets natively, so you don’t even set dataset_text_field — TRL detects the columns. The field still exists on SFTConfig for the single-text-column case; we just don’t need it.

The knobs that matter: hyperparameters and the loss curve

You have a working SFTConfig. Now: what do its numbers do, and how do you tell a good run from a bad one by looking at the loss?

The hyperparameters, and what each one moves

• learning_rate — the size of each update step. For SFT the practitioner range is roughly 1e-5 to 2e-4 (full fine-tuning sits toward the low end; adapters tolerate the high end — Anvil’s default of 2e-4 anticipates Module 4’s LoRA). Too high and the loss oscillates or diverges; too low and it barely moves (a “flat loss”). Treat these as starting heuristics, not constants — they depend on your data and model.
• num_train_epochs — how many passes over the data. For SFT, 1 to 3 is typical. I cap it at 3 on a clean dataset; past that the model starts memorizing demonstrations and your eval loss climbs even as train loss keeps falling. That divergence is the definition of overfitting, and it’s why train loss alone is not a quality signal.
• per_device_train_batch_size and gradient_accumulation_steps — the effective batch size is per_device_batch_size × gradient_accumulation_steps × num_gpus. Accumulation simulates a large batch when VRAM can’t hold one: it runs several micro-batches, summing their gradients, and only then takes one optimizer step. Anvil uses 8 × 2 = 16 per GPU. When you hit out-of-memory errors, this pair (plus the techniques in Module 5) is your lever.
• max_length — caps sequence length. Too short truncates long examples (the model never sees the full call); too long wastes VRAM on padding. 1024 fits Anvil’s schemas-plus-call.

Reading a loss curve

The loss curve is your primary diagnostic. There are four profiles, and each has a likely cause and a fix:

flowchart LR
    subgraph H["Healthy descent"]
        h["loss drops fast,<br/>then flattens;<br/>eval follows train"]
    end
    subgraph F["Flat loss"]
        f["loss barely moves<br/>→ lr too low, OR the chat<br/>template was never applied<br/>(model never sees the format),<br/>OR loss on the wrong tokens"]
    end
    subgraph D["Diverging loss"]
        d["loss climbs / spikes / NaN<br/>→ lr too high, or numeric<br/>instability (dtype)"]
    end
    subgraph O["Overfitting"]
        o["train loss falls but<br/>EVAL loss rises<br/>→ too many epochs,<br/>dataset too small"]
    end
    H -->|nothing — ship it| H2["✅"]
    F -->|raise lr; verify data.py applied the template| F2["fix data/lr"]
    D -->|lower lr; check bf16/dtype| D2["fix lr"]
    O -->|fewer epochs; more / cleaner data| O2["fix epochs/data"]
    style H2 fill:#d4edda

The one that catches everyone is the flat loss. A beginner’s instinct is “train for more epochs.” That almost never helps, because a flat loss usually means the model isn’t seeing what you think it’s seeing — most often the chat template wasn’t applied, so the model is staring at raw JSON instead of the <tool_call> format it’s supposed to learn (this is why Module 2’s apply_chat_template discipline matters so much), or the learning rate is too low to move anything. Check data.py and the rendered prompt first; raise the lr second. More epochs last, if ever.

Full fine-tuning vs PEFT

Anvil v0 here is a full fine-tune in concept — peft_config=None updates all the weights. That’s the right place to introduce the distinction that drives the next two modules.

Full fine-tuning updates every weight. That’s maximum adaptation capacity, but it’s brutal on VRAM: you store the weights plus the gradients plus the optimizer states (Adam keeps two moments per parameter), which together run to several times the model’s size. A 4B model in full FP/BF16 full fine-tuning does not fit a free 16 GB T4 — which is exactly why this lab trains a subsample on the free tier and frames the full-scale full FT as conceptual. The free-GPU happy path is PEFT, starting next module.

PEFT (parameter-efficient fine-tuning) freezes the base model and trains a tiny number of new parameters — adapters — typically around 1% of the weights. The optimizer only tracks those, so the VRAM footprint collapses and a 4B fine-tune fits a consumer GPU. This is the course’s default from Module 4 on (LoRA), made even leaner in Module 5 (QLoRA).

	Full fine-tuning	PEFT (LoRA/QLoRA, M4–M5)
Params trained	All of them (100%)	~1% (adapter weights)
VRAM	Heavy (weights + grads + optimizer states ≈ several× model size)	Low (optimizer tracks ~1%)
Quality / capacity	Maximum adaptation	Very close on most tasks; ceiling slightly lower
Artifact size	A full model copy (GBs)	A small adapter (MBs)
Swappable?	No — one monolithic checkpoint	Yes — load/unload adapters on one base
When to use	You have the VRAM and need maximum capacity	Default for consumer GPUs (the rest of this course)

The mechanics of LoRA — LoraConfig, r, lora_alpha, target_modules — are Module 4. Here it’s the framing: full FT is the concept; PEFT is how you actually do this on a free GPU.

Forging Anvil v0: the factory, the run, and before/after

Time to build. Three pieces: the factory (frozen here), the run, and the proof.

anvil/config.py — the model/tokenizer factory (frozen)

Every model choice — which model, what dtype, and later which quantization and which adapters — lives in one place: get_model_and_tokenizer(tier). No script hard-codes a model ID. The payoff is reproducibility (one place sets the seed of every run) and swappability (changing the base model is a one-line edit). This is the frozen contract from the fil-rouge spec; its signature is stable for Modules 4 through 15.

MODEL_ID = "Qwen/Qwen3-4B-Base"                 # Apache-2.0, base (we teach the behavior ourselves)
FALLBACK_MODEL_ID = "HuggingFaceTB/SmolLM3-3B"  # Apache-2.0 alternative


def get_model_and_tokenizer(tier: str = "full"):
    """Return (model, tokenizer) for a tier. Module 3 supports 'full' and 'tiny'."""
    import torch
    from transformers import AutoModelForCausalLM, AutoTokenizer

    if tier == "tiny":      # the offline test fixture (CPU, no network)
        model_id, kwargs, local = _tiny_dir(), {"dtype": torch.float32}, True
    elif tier == "full":    # full-precision base — Module 3's full SFT
        model_id, kwargs, local = MODEL_ID, {"dtype": torch.bfloat16}, False
    else:
        raise ValueError(f"Unknown tier {tier!r} (module 3 supports 'full'/'tiny'; "
                         "'qlora' arrives in module 5, 'unsloth' in module 6)")

    tokenizer = AutoTokenizer.from_pretrained(model_id, local_files_only=local)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token
    model = AutoModelForCausalLM.from_pretrained(model_id, local_files_only=local, **kwargs)
    return model, tokenizer

Three things to notice. dtype= (not the deprecated torch_dtype=) sets BF16 on the GPU. We always set a pad_token because base models often ship none, and batching/collation break without one. And tier is the extension point: "full" and "tiny" exist today; "qlora" arrives in Module 5 and "unsloth" in Module 6. The "tiny" tier is the secret behind the offline test — it loads a 160 KB tiny-Qwen3 fixture (carrying the real Qwen3 tool-calling chat template) with local_files_only=True, so the SFT loop runs on CPU with no network.

anvil/train_sft.py — the run

The training function is small on purpose, and it carries Anvil through Modules 3–6 by varying tier and use_lora. Module 3 is tier="full", use_lora=False:

def train_sft(*, tier: str = "full", use_lora: bool = False, data_limit: int | None = None,
              output_dir: str = "outputs/anvil-sft", **arg_overrides):
    """Fine-tune Anvil with SFT and return the (trained) SFTTrainer."""
    from trl import SFTTrainer

    model, tokenizer = config.get_model_and_tokenizer(tier)
    train_dataset = data.load_sft_dataset("train", tokenizer, limit=data_limit)
    args = config.sft_args(output_dir=output_dir, **arg_overrides)

    peft_config = None
    if use_lora:
        if not hasattr(config, "lora_config"):
            raise RuntimeError("use_lora=True needs config.lora_config() — that arrives in Module 4")
        peft_config = config.lora_config()

    trainer = SFTTrainer(
        model=model,
        args=args,
        train_dataset=train_dataset,
        processing_class=tokenizer,   # TRL v1: processing_class, never tokenizer=
        peft_config=peft_config,
    )
    trainer.train()
    trainer.save_model(output_dir)
    return trainer

Notice what’s not there: no tokenizer=, no max_seq_length, no dataset fields on the Trainer. The config comes from sft_args(), the data from data.load_sft_dataset(...) (which returns the prompt/completion columns), the tokenizer goes in processing_class. use_lora=True deliberately raises a clear error pointing at Module 4 — the LoRA path is real, just not yet wired. Run it from the repo root:

(One harmless log to expect on the run: trl prints a FutureWarning that the loss_type default will change from nll to chunked_nll in TRL 1.7. It’s just a forward-compatibility heads-up — nothing is wrong, the run behaves identically on 1.6.0, so don’t let it startle you.)

uv run --env-file .env python -m anvil.train_sft

Before and after — the proof

This is the payoff moment. Take one tool-call prompt — the same one for both models — render it with data.render_prompt(...) (chat template, add_generation_prompt=True), and generate with the base model, then with Anvil v0:

=== BASE (Qwen3-4B-Base) ===
Sure! To get the weather you would call the weather API. Here is how:
the get_weather function takes a city... {"city": "Paris", maybe also units}

=== ANVIL v0 (after SFT) ===
<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris"}}
</tool_call>

The base model rambles in prose and never produces a clean call; Anvil v0 emits a valid <tool_call> block in exactly the shape the chat template defines. That contrast is the whole module made visible.

One honest caveat, because it’s the senior-engineer move: this before/after is anecdotal — a single example. A falling loss curve plus one good generation proves training ran and moved the model; it does not prove the model is reliably better. Module 7 replaces this qualitative peek with task-grounded evaluation — JSON validity, function-name accuracy, argument match — on a held-out set, base vs fine-tuned. Here, you just get to see it work. We measure it properly later.

Hands-on lab: build Anvil v0

Goal: freeze the model/tokenizer factory, run an SFT through SFTTrainer/SFTConfig (TRL v1 idioms), save the checkpoint, and show the before/after generation plus a falling loss curve.

What you’ll see: uv run --env-file .env python -m anvil.train_sft logs a loss that descends, saves a checkpoint to outputs/anvil-sft/, and (in the notebook) prints the base-vs-Anvil-v0 generation. The offline smoke test runs the whole SFT loop on CPU in seconds.

The full, tested code lives in finetune-prod-labs/module-03. Its core tests pass offline — no GPU, no token, no network — so you can verify the plumbing before you spend a minute of GPU time.

Step 1 — Set up. Copy the Module 2 state forward (the repo is cumulative). The labs share one root pyproject.toml and committed uv.lock; the locked core is all you need for SFT — trl==1.6.0, transformers==5.12.0, peft==0.19.1, datasets==5.0.0, accelerate==1.14.0, torch==2.12.0. (bitsandbytes is not needed here — that’s Module 5’s QLoRA.) Copy .env.example to .env and set your HF_TOKEN; never commit .env.

Step 2 — Write the frozen factory (anvil/config.py). MODEL_ID = "Qwen/Qwen3-4B-Base" (fallback HuggingFaceTB/SmolLM3-3B). get_model_and_tokenizer(tier) loads the model and tokenizer for tier="full" (BF16 on GPU) or tier="tiny" (the offline CPU fixture), always sets a pad_token, and is the only place a model is loaded. The sft_args() helper builds the SFTConfig with max_length, completion_only_loss=True, and the hyperparameters. This file is frozen — Modules 4–15 reuse its signature.

Step 3 — Run SFT (anvil/train_sft.py). train_sft(tier="full", use_lora=False) loads (model, tokenizer), loads the train split via data.load_sft_dataset(...), builds the config, instantiates SFTTrainer(model, args=cfg, train_dataset=..., processing_class=tokenizer, peft_config=None), trains, and saves. Run it on a T4:

uv run --env-file .env python -m anvil.train_sft

Step 4 — See the before/after. Render one tool-call prompt with data.render_prompt(row, tokenizer), generate with the base model and then with Anvil v0, and print both. Base: broken prose. Anvil v0: a valid <tool_call>. (The notebook does this side by side.)

Step 5 — Plot the loss. Pull trainer.state.log_history, plot the loss against step, and save a PNG. You’re looking for the healthy-descent profile from the diagram. The lab keeps report_to=[] (no W&B) so it stays offline-friendly; flip on Weights & Biases by setting report_to=["wandb"] and WANDB_API_KEY if you want live curves.

Step 6 — Run the offline tests. The smoke test loads the tier="tiny" fixture, builds SFTConfig/SFTTrainer with processing_class=, runs 2 real steps over 8 examples, and asserts the loss is finite and save_model wrote a directory — all on CPU, deterministic (seed=0), no network:

uv run pytest module-03/tests/

Try it yourself:

1. Vary learning_rate — try 5e-6 and 5e-4 — and watch the loss curve. One should go nearly flat, the other should oscillate or climb. Match what you see to the diagnostic diagram.
2. Toggle completion_only_loss=False then True, retrain, and compare the before/after generation. You’re watching what prompt-masking does to what the model actually learns.

What this lab does NOT do, on purpose. No LoRA/PEFT (Module 4) — peft_config=None; we only name the slot and frame the full-vs-PEFT tradeoff. No QLoRA, bitsandbytes, or 4-bit (Module 5). No Unsloth/Axolotl (Module 6) — tier="unsloth" isn’t implemented yet. No task-grounded eval (Module 7) — the before/after is qualitative. No DPO/GRPO (Modules 8/10), no Hub push (Module 14).

In production

What changes outside this lab is mostly scale and discipline. A production SFT runs on thousands to millions of demonstrations, not a 1,500-row fixture, and often as full fine-tuning across multiple GPUs (driven by accelerate) — which is exactly why teams reach for PEFT (Modules 4–5) the moment they’re on a single consumer card. You always fix the seed (SFTConfig(seed=...)) so a run is reproducible, and you log to TensorBoard or Weights & Biases (report_to=[...]) so you can read the loss live instead of after the fact.

The non-negotiable: keep a held-out set you never train on, and watch its eval loss. A training loss that falls tells you the optimizer is working; it says nothing about whether the model generalizes. I cap SFT at 1–3 epochs on a clean dataset — past that you’re memorizing demonstrations, and the eval loss will tell you so by climbing. And the number-one production failure mode is upstream of all of this: mis-formatted data — a forgotten chat template — which gives you a flat loss and a model that never learns the format. That’s why Module 2 exists, and why “check data.py first” is the right reflex.

The teaser for what’s coming: a falling loss is “training ran.” Module 7 is what turns that into “the model is measurably better on its task.” Those are not the same claim, and conflating them is how people ship fine-tunes that look trained and aren’t.

Concept check

Why this matters. This module gives you the four skills you’ll use in every training module after it: knowing what SFT optimizes (next-token loss on demonstrations), placing arguments correctly on the Trainer vs the Config (the TRL v1 idioms), choosing hyperparameters with intent, and diagnosing a loss curve. The factory and idioms you set up here are reused, unchanged, all the way to the capstone. The questions below stay inside what this module taught.

1. A developer wants to cap sequence length at 1024 and point the trainer at their text column, and they need to pass the tokenizer. Where does each of these go?

   • A. All three on SFTTrainer: max_length=, dataset_text_field=, tokenizer=.
   • B. max_length and dataset_text_field on SFTConfig (via args=); the tokenizer in SFTTrainer(processing_class=...).
   • C. max_seq_length and dataset_text_field on SFTConfig; tokenizer= on SFTTrainer.
   • D. All three on SFTConfig, including the tokenizer.

2. You train for 3 epochs and the train loss barely moves off its starting value — a nearly flat line. What’s the most likely cause and fix?

   • A. The model is overfitting; train for fewer epochs.
   • B. The learning rate is too low or the chat template was never applied (the model never sees the target format); check data.py/the rendered prompt and raise the lr.
   • C. The learning rate is too high; lower it.
   • D. The dataset is too large; subsample it.

3. A teammate says “we’ll SFT the model on our internal docs so it knows our product facts.” What’s the problem with that framing?

   • A. Nothing — SFT is exactly how you inject facts.
   • B. SFT minimizes next-token loss on demonstrations: it changes behavior and format, not facts. Facts are a retrieval (RAG) problem; SFT on docs teaches style, not reliable recall.
   • C. SFT can inject facts, but only with completion_only_loss=True.
   • D. SFT injects facts only if you train for 10+ epochs.

4. A 4B model in full fine-tuning won’t fit your free 16 GB T4 — you OOM immediately. What’s the right move, and why?

   • A. Lower max_length to 128 and keep full fine-tuning.
   • B. Use PEFT (LoRA): train ~1% of the weights as a swappable adapter, because full FT’s optimizer states and gradients are what blow up VRAM.
   • C. Switch to PPO, which uses less memory.
   • D. Increase gradient_accumulation_steps until it fits — full FT will then run in 16 GB.

5. You copy a 2024 snippet that passes tokenizer=tok and max_seq_length=1024 to SFTTrainer, and it raises a TypeError. What are the exact corrections?

   • A. Use processing_class=tok; move max_length onto SFTConfig. Both keywords were removed (v0.16 and v0.20).
   • B. Pin trl==1.0 — the snippet needs the old major version.
   • C. Pass tokenizer=tok to SFTConfig instead of the Trainer.
   • D. Downgrade to transformers==4.x so the old keywords still work.

Answers.

1. B. The tokenizer always goes in processing_class= on the SFTTrainer (never tokenizer=). max_length and dataset_text_field are fields of SFTConfig, passed via args=. Option C uses the removed max_seq_length; A and D misplace the dataset/length fields and the tokenizer.
2. B. A flat loss means the model isn’t learning the target — almost always because the format never reached it (chat template not applied) or the step size is too small. Check the rendered prompt and data.py, then raise the learning rate. More epochs (A is the opposite problem; D is unrelated) won’t fix a model that can’t see its target.
3. B. SFT reweights token probabilities toward the demonstrated behavior and format — it’s the same objective as pretraining, on different data. It does not give you reliable factual recall; that’s what retrieval (RAG, Module 1’s escalation ladder) is for. Treating SFT as fact injection is the classic wrong-tool mistake.
4. B. PEFT trains a tiny adapter (~1% of the weights), so the optimizer only tracks those parameters — and it’s the optimizer states plus gradients of all the weights that make full FT explode past 16 GB. Lowering max_length (A) helps marginally but doesn’t change the weight count; PPO (C) is unrelated and not the 2026 default; accumulation (D) reduces activation memory, not the full-FT weight/optimizer footprint.
5. A. tokenizer= was removed in trl v0.16 (use processing_class=tok) and max_seq_length in v0.20 (use max_length on SFTConfig). “TRL v1” is the shipped 1.x line on PyPI (1.6.0 as of June 2026), so B’s trl==1.0 is both wrong and unnecessary. C misplaces the tokenizer; D pins a transformers line the rest of the stack won’t run on.

Common pitfalls.

• Passing tokenizer= / max_seq_length= / dataset fields to the Trainer. As of trl 1.6.0, June 2026: the tokenizer goes in processing_class= (Trainer); dataset_text_field/packing/completion_only_loss are fields of SFTConfig; max_seq_length no longer exists (it’s max_length). ~70% of tutorials still show the dead signature — check against the installed version. (And “TRL v1” now is the major version on PyPI, version 1.6.0 — the old “never trl==1.0” advice is obsolete; just pin the exact version you tested.)
• “SFT injects facts.” No — it adjusts behavior and format by demonstration (next-token loss). Facts are a retrieval problem (RAG, Module 1). Confusing the two leads to the wrong escalation and a model that confidently hallucinates.
• Mistaking a falling loss for “the model is better.” A descending train loss proves training ran, not that the model generalizes. Keep a held-out eval set, watch its loss, and remember the real quality signal is task-grounded evaluation (Module 7).

Key takeaways

• SFT is next-token loss on demonstrations — the same causal cross-entropy objective as pretraining, only the data changed. It moves behavior and format, not facts (InstructGPT, arXiv 2203.02155, made SFT the first stage of the pipeline).
• The tokenizer goes in processing_class=, never tokenizer= (removed in trl v0.16).
• dataset_text_field, max_length, packing, and completion_only_loss are fields of SFTConfig, passed via args= — not Trainer arguments. The Trainer gets the model, the data, the tokenizer, and peft_config.
• It’s max_length, not max_seq_length (removed in v0.20 → TypeError); the default is 1024.
• Read the loss curve: flat → raise the lr or check the chat template was applied; diverging → lower the lr; eval loss rising while train falls → overfitting (fewer epochs, more data).
• Full FT vs PEFT: full updates all weights (heavy VRAM); PEFT trains ~1% as a swappable adapter and is the course default from Module 4. Use completion_only_loss=True to learn the answer, not the prompt.
• The anvil/config.py factory centralizes model loading for the entire course — get_model_and_tokenizer(tier), one MODEL_ID, frozen here and reused through the capstone.

What’s next

Anvil v0 works — but full fine-tuning a 4B model is heavy, and it won’t fit a free GPU at full scale. In Module 4: LoRA you’ll train 1% of the weights: the same SFTTrainer, one peft_config, a swappable adapter measured in megabytes that fits a T4 with room to spare. That’s where the free-GPU path really begins.

Want the full fine-tuning concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · ← Module 2: Data Is the Model · Module 4: LoRA — Train 1% of the Weights → · Lab code for this module

References

• InstructGPT — “Training language models to follow instructions with human feedback” (the paper that put SFT first in the pipeline): https://arxiv.org/abs/2203.02155
• TRL SFTTrainer documentation (the processing_class/max_length/SFTConfig idioms): https://huggingface.co/docs/trl/main/en/sft_trainer
• TRL SFTConfig API reference: https://huggingface.co/docs/trl/main/en/sft_trainer#trl.SFTConfig
• transformers TrainingArguments (the base class SFTConfig inherits from; processing_class on Trainer): https://huggingface.co/docs/transformers/main/en/main_classes/trainer
• Qwen3 model family (the Apache-2.0 base model and its tool-calling chat template): https://huggingface.co/Qwen/Qwen3-4B-Base