aifromzero.dev

Serve It: vLLM, Ollama, and TGI (Fine-Tuning in Production, Module 13)

Module 13 of 15 15 min read Lab code ↗

This is Module 13 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Start at Module 1 or browse the full syllabus.

A model on disk is not a service

By the end of Module 12, Anvil is finished in every sense a training notebook understands. It’s been supervise-fine-tuned, given a LoRA adapter, aligned with DPO, pushed by GRPO, merged, and quantized to a GGUF file. You can load it and it answers. So you do the obvious thing: you write model.generate() in a for loop and call it a day.

That works for exactly one afternoon. The moment anyone else needs to call Anvil — a teammate, a web app, a smoke test in CI — the loop falls apart. There’s no HTTP endpoint. There’s no way to measure how long a request takes. There’s no concurrency, no model staying warm in memory, nothing reusable. A for loop over generate() is a demo, not a service.

And here’s the part that trips people up even when they reach for a real server: the right server depends on the artifact you’re holding. You finished Module 12 with two different things — a tiny LoRA adapter that only lives on top of Qwen3-4B-Base, and a standalone merged GGUF file. They are not interchangeable, and the server that serves one cannot serve the other. Plenty of engineers merge when they shouldn’t (throwing away the cheapest win in serving), or try to feed a raw adapter to Ollama and watch it choke. This module serves Anvil twice — vLLM over the adapter (no merge), Ollama over the GGUF — calls each with a real tool-call prompt, measures the latency, and draws the line between “serving foundations” and the scaling work that belongs to a future course called Turbine.

In this module

You’ll learn how to:

  1. Serve the LoRA adapter directly with vLLM (--enable-lora + --lora-modules), hit an OpenAI-compatible endpoint, and target the adapter per request by name — without merging.
  2. Serve the merged GGUF from Module 12 with Ollama (a Modelfile FROM ./anvil.gguf, a local REST API on :11434) — a no-GPU, CPU / Apple-Silicon path.
  3. Choose the right server for the artifact you have: adapter (vLLM) vs merged GGUF (Ollama) vs multi-LoRA at scale (TGI, LORA_ADAPTERS) — what each serves, on what hardware, and when to reach for which.
  4. Call the endpoint with a real tool-call prompt and measure latency honestly — mean / p50 / p95 over repeated calls, plus the concept of TTFT (time-to-first-token) — numbers, not vibes.
  5. Recognize the runtime-LoRA security risk: dynamic adapter loading via VLLM_ALLOW_RUNTIME_LORA_UPDATING + /v1/load_lora_adapter is something vLLM itself warns not to expose on an untrusted endpoint — and understand why this is a serving foundations module, not a serving-at-scale one (that’s Turbine).

You’ll build: Anvil becomes an endpoint. anvil/serve.py serves Anvil two ways — vLLM with --enable-lora over your Module-4/5 adapter (no merge), and Ollama over the Module-12 merged GGUF — and exposes one OpenAI-shaped tool-call request that works against either. You’ll send the same prompt to both, confirm the tool-call JSON is valid, and measure latency. The smoke test mocks the server so the whole thing stays offline and GPU-free; the real serving runs in the notebook.

Theory areas covered:

  • T11 — Inference quantization & serving — 7% of the course. This module owns the serving half of T11; the quantization half (PTQ, GGUF, AWQ, GPTQ, merge_and_unload) is Module 12’s, and we reuse its output rather than re-teach it.

This module builds on T11-quant (the merged GGUF from Module 12, merge_and_unload), on T4/T5 (the LoRA/QLoRA adapter you trained and aligned), and on T2 (the chat template that formats messages + tools exactly as during training).

Prerequisites: Modules 1–12 — especially the trained-and-aligned LoRA adapter (M4/M5, sharpened by M8–M10) and the merged GGUF (M12). For the lab: for vLLM, an NVIDIA CUDA GPU (a free Colab/Kaggle T4 with 16 GB is enough for Qwen3-4B + an adapter); for Ollama, no GPU at all (CPU / Apple Silicon via the GGUF). HF_TOKEN set in your environment (M1) if you pull the base from the Hub. The smoke test launches neither vLLM nor Ollama — it mocks the server.

Where you are

Anvil’s pipeline, with this module marked:

  • ✅ M1–M2 — baseline + data (chat template, tools=).
  • ✅ M3–M6 — SFT, LoRA, QLoRA (fits a free 16 GB T4), Unsloth/Axolotl.
  • ✅ M7 — task-grounded eval (tool_call_valid_json, function_name_accuracy, argument_match).
  • ✅ M8–M10 — preference alignment (DPO/ORPO/KTO/SimPO) and GRPO + verifiable rewards.
  • ✅ M11–M12 — merge (mergekit, merge_and_unload) and quantize (GGUF / AWQ / GPTQ).
  • 👉 M13 — Serve It: vLLM, Ollama, and TGI. Anvil stops being a training artifact and becomes an endpoint you call.
  • ⬜ M14–M15 — publish to the Hub (model card, base_model:), capstone.

The one-line before/after for Anvil: it goes from “trained, aligned, merged, and quantized to a GGUF — but never actually served” to “served two ways (vLLM over the adapter, Ollama over the merged GGUF) and called as a real endpoint, with measured latency.” serve.py is the last technical link before publishing (M14 puts Anvil on the Hub) and is replayed in the capstone (M15). And it’s the on-ramp to a future course, Turbine, which takes serving deep — continuous batching, paged KV-cache, autoscaling. This module lays the foundations; Turbine builds the skyscraper.

From artifact to endpoint: what you’re actually serving

Before a single command, get this straight, because everything downstream hangs on it. Module 12 left you with two servable artifacts, and they are fundamentally different objects.

The first is the LoRA adapter — a small directory (adapter_config.json plus a few megabytes of low-rank weights) produced in Modules 4–5 and sharpened by DPO/GRPO. An adapter is not a model. It’s a patch that only means anything applied on top of Qwen/Qwen3-4B-Base. On its own it can’t generate a token.

The second is the merged GGUF — a single self-contained file. To build it (Module 12) you first folded the adapter back into the base with merge_and_unload, producing a full-precision standalone model, then quantized it to GGUF with llama.cpp. The GGUF is a complete model; the adapter has dissolved into the base weights.

The key fact of this module: the server you choose is determined by the artifact you hold.

  • vLLM serves the adapter as-is. It loads Qwen3-4B-Base once and applies the adapter per request — no merge needed. You hand it the adapter directory.
  • Ollama (and llama.cpp under it) serve a GGUF — which means an already-merged model. You hand it the GGUF file. You cannot serve a raw adapter through Ollama; there’s nothing for it to attach the adapter to.

That last point is a hard rule, not a preference. A GGUF is a merged model by construction, so a raw LoRA adapter cannot be served as a GGUF — you must merge_and_unload first (Module 12). The llama.cpp GGUF converter needs full weights; it has no concept of “base + adapter.”

Why serving the adapter without merging is often the right call

Here’s the “why” that makes adapter-serving more than a convenience. An adapter weighs a few megabytes. The base weighs gigabytes. So you can load the base once into GPU memory and stack many adapters on top of it, routing each request to the right one by name. Deploy the base once, serve many specialists. Ten fine-tuned variants of Anvil don’t cost ten models’ worth of VRAM — they cost one base plus ten small adapters. That’s the multi-LoRA argument, and it’s why vLLM and TGI exist in this conversation.

You merge only when you want a standalone artifact: a GGUF for Ollama / CPU / Apple Silicon, an air-gapped binary, or a frozen model repo. Merging gives you zero adapter-application overhead and a single file — at the cost of throwing away the share-one-base economics. So the decision is not “adapter vs merged, which is better”; it’s “do I want many cheap specialists on one base, or one self-contained binary?”

Here are the two serving paths, side by side:

flowchart TB
    A["Anvil (trained + aligned)"]

    A --> AD["LoRA adapter (M4/M5)<br/>~MB, lives ON the base"]
    AD --> V["vLLM --enable-lora<br/>--lora-modules anvil=./adapter<br/>(base loaded ONCE, adapter applied per request, NO merge)"]
    V --> VE["OpenAI-compatible endpoint :8000<br/>model: 'anvil'"]

    A --> M["merge_and_unload (M12)"]
    M --> MB["merged base (full precision)"]
    MB --> GG["GGUF (M12, llama.cpp q4_k_m)"]
    GG --> O["Ollama (Modelfile FROM ./anvil.gguf)"]
    O --> OE["local REST endpoint :11434"]

    note["vLLM serves the *adapter*; Ollama serves the *merged GGUF* — not interchangeable"]:::n -.-> A
    classDef n fill:#fff,stroke:#bbb,stroke-dasharray:5 5,color:#888;

One framing note before the commands: this module teaches which server for which artifact and how to call and measure it. The scaling — continuous batching, paged KV-cache, speculative decoding, autoscaling, multi-GPU — is the Turbine course. We don’t touch it here, and we won’t pretend to.

Serving the adapter with vLLM

vLLM is a high-throughput inference server for LLMs. The feature that matters for a fine-tuned model is that it can serve LoRA adapters directly, without merging them into the base.

Launching the server

You start vLLM from the command line. The lab builds the exact command for you with serve.vllm_serve_command, so the flags are pinned and you never fat-finger them. Here’s the function — note that it only adds the LoRA flags when you give it an adapter_path:

def vllm_serve_command(base_model: str, *, adapter_path: str | None = None,
                       max_lora_rank: int = 16, port: int = 8000) -> list[str]:
    """Build a `vllm serve` command. With `adapter_path` it serves the LoRA adapter
    under the name `anvil` (no merge); without it, it serves merged/base weights."""
    cmd = ["vllm", "serve", base_model, "--port", str(port)]
    if adapter_path:
        cmd += ["--enable-lora", "--lora-modules", f"anvil={adapter_path}",
                "--max-lora-rank", str(max_lora_rank)]
    return cmd

Rendered for Anvil’s DPO adapter, that’s:

vllm serve Qwen/Qwen3-4B-Base --port 8000 \
  --enable-lora \
  --lora-modules anvil=outputs/anvil-dpo \
  --max-lora-rank 16

Read each flag at the “why” level, not just the “what” (as of vLLM 0.23.0, June 2026 — vLLM ships often; re-check the flag names against the docs at your pinned version):

  • --enable-lora turns on LoRA support in the engine. Without it, --lora-modules is silently ignored — the single most common “why isn’t my adapter loading” bug.
  • --lora-modules anvil=outputs/anvil-dpo registers the adapter directory under a name (anvil). The base is loaded into VRAM once; the adapter is applied on the fly. You can pass several name=path pairs here — that’s multi-LoRA on one base.
  • --max-lora-rank 16 must be ≥ the rank of your adapter. Anvil’s LoRA is r=16 (Module 4’s lora_config()), so 16 fits exactly. If you trained at a higher rank, raise this; if it’s too low, vLLM refuses the adapter.

vLLM needs an NVIDIA CUDA GPU. For Anvil — a 4B base plus one small adapter — a free 16 GB T4 (Colab/Kaggle) is plenty. There is no CPU path for vLLM here; CPU serving is Ollama’s job.

Calling the endpoint

vLLM exposes an OpenAI-compatible server, so any OpenAI client works against it — you just point base_url at it and set a throwaway key. The crucial detail: the model field selects the adapter by its registered name. The lab’s build_tool_call_request constructs the payload; model defaults to "anvil":

def build_tool_call_request(query: str, tools: list[dict], *, model: str = "anvil",
                            temperature: float = 0.0) -> dict:
    """An OpenAI-compatible chat-completions payload for a tool-call request
    (works on vLLM / Ollama / TGI)."""
    return {
        "model": model,                 # <- the adapter name routes the request
        "messages": [{"role": "user", "content": query}],
        "tools": tools,
        "temperature": temperature,
    }

You POST that to /v1/chat/completions. With the OpenAI SDK it looks like the snippet everyone already knows — client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY"), then client.chat.completions.create(model="anvil", messages=..., tools=...). The lab keeps it framework-light with serve.call_endpoint, a thin requests.post to the same path:

def call_endpoint(base_url: str, request: dict, *, timeout: float = 30.0) -> dict:
    """POST a chat-completions request to a running server, return the JSON response."""
    import requests
    resp = requests.post(f"{base_url.rstrip('/')}/v1/chat/completions",
                         json=request, timeout=timeout)
    resp.raise_for_status()
    return resp.json()

The “why”: it’s the same base being served; the model field routes the request to the right adapter. That routing is exactly what makes multi-LoRA possible — change model to "other-adapter" and the same server, the same loaded base, answers with a different specialist. And because the server speaks the OpenAI shape, the same tools schema and the same response parsing you wrote in earlier modules just work.

⚠️ Loading adapters at runtime is a security risk

vLLM can also load adapters at runtime — without restarting the server — if you set VLLM_ALLOW_RUNTIME_LORA_UPDATING=1 and call the /v1/load_lora_adapter endpoint with a path or Hub URL. Convenient in a private dev box or CI: hot-swap a freshly trained adapter without a restart.

vLLM itself warns that this is a security risk. That endpoint lets any caller load arbitrary weights from an arbitrary path or URL into your running server. On an untrusted, public, or production endpoint that’s a remote-code/arbitrary-load vector — so don’t enable it on anything exposed. Keep it to internal dev/CI, behind your own network boundary. This is a differentiating point most serving tutorials skip, and it shows up in the misconception box and the concept check below. As of vLLM 0.23.0, June 2026. The lab never enables it.

Serving the merged GGUF with Ollama (and multi-LoRA with TGI)

vLLM needs a GPU. Ollama doesn’t — and that’s the whole point of the second path.

Ollama: serve the merged GGUF

Ollama runs GGUF files (llama.cpp under the hood) on CPU or Apple Silicon, with no NVIDIA card required. You point it at the merged GGUF from Module 12 via a Modelfile — a tiny declaration of which weights to load and how to format turns. The lab renders one with serve.ollama_modelfile:

def ollama_modelfile(gguf_path: str, *, stop: str = "<|im_end|>") -> str:
    """Render an Ollama Modelfile that loads Anvil's merged GGUF (Module 12)
    and stops on the turn end."""
    return (
        f"FROM {gguf_path}\n"
        f'PARAMETER stop "{stop}"\n'
        "PARAMETER temperature 0\n"
    )

For Anvil’s quantized GGUF that produces:

FROM ./anvil-q4_k_m.gguf
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0

Then ollama create anvil -f Modelfile registers it, and Ollama serves a local REST API on :11434 (/api/chat, /api/generate, and an OpenAI-compatible shape too). The “why” behind each line:

  • FROM ./anvil-q4_k_m.gguf loads a GGUF — an already-merged model (Module 12). You never hand Ollama a raw adapter; there’s nothing to attach it to.
  • No GPU required. This is the CPU / Apple-Silicon path. Ideal for testing Anvil locally, demoing on a laptop, or serving on a machine with no NVIDIA card.
  • PARAMETER stop "<|im_end|>" is the chat-template detail that bites people. Qwen3 closes each turn with <|im_end|>; if the server doesn’t stop there, Anvil keeps generating past its answer. The GGUF should embed Qwen3’s chat template so messages + tools are formatted exactly as in training (Module 2), and the Modelfile pins the stop token to match. Set temperature 0 for deterministic, reproducible tool calls.

The request → engine → response path

The beautiful part of building on the OpenAI shape: the same build_tool_call_request payload and the same validation work against both engines. Only the engine — and the regime — changes.

flowchart TB
    P["tool-call prompt<br/>(user message + tool schemas)"]
    P -->|t0| V["vLLM<br/>(base + adapter, GPU)"]
    P -->|t0| O["Ollama<br/>(merged GGUF, CPU / Apple Silicon)"]
    V -->|first token = TTFT| VG["generate"]
    O -->|first token = TTFT| OG["generate"]
    VG -->|response complete = total latency| R["tool-call JSON"]
    OG -->|response complete = total latency| R
    R --> CHK{"valid JSON?<br/>right tool name?"}
    CHK -->|yes| OK["✅ served correctly"]
    CHK -->|no| BAD["❌ regression"]
    note["same prompt, same validation —<br/>only the engine changes"]:::n -.-> P
    classDef n fill:#fff,stroke:#bbb,stroke-dasharray:5 5,color:#888;

To confirm the served output is correct, you reuse Anvil’s Module-7 parser — eval.parse_tool_calls and function_name_accuracy — as a serving smoke check: did the endpoint return a parseable tool call with the right name? That’s not the formal task-grounded eval of Module 7 (this is a “did serving work” sanity check, not a benchmark), but it reuses the same notion of “a correct call.”

TGI multi-LoRA, in a survey

Text Generation Inference (TGI) is Hugging Face’s production server, and its angle here is multi-LoRA at scale. It serves many adapters on one base via the LORA_ADAPTERS environment variable — the pitch is “deploy the base once, serve 30+ adapters.” The lab builds that env string but does not deploy TGI (that’s Turbine territory):

def tgi_env(adapter_paths: dict[str, str]) -> dict[str, str]:
    """The env for TGI multi-LoRA serving: LORA_ADAPTERS=name=path,name2=path2."""
    return {"LORA_ADAPTERS": ",".join(f"{name}={path}" for name, path in adapter_paths.items())}

So tgi_env({"anvil-sft": "/m/sft", "anvil-dpo": "/m/dpo"}) yields LORA_ADAPTERS=anvil-sft=/m/sft,anvil-dpo=/m/dpo. The “deploy once, serve 30+” claim and the exact env contract are version-dependent — as of June 2026, re-check against the TGI docs. The takeaway is the lever, not the deployment: vLLM and TGI share one base in memory and route by adapter, so serving N specialists costs about one model plus N small adapters. That’s the economic argument for not merging when you have many variants — and the reason “merge everything before serving” is so often the wrong reflex.

Measuring what matters: latency (and choosing a server)

This course measures instead of guessing, and serving is where that habit pays off. Define the metrics at the “why” level:

  • TTFT (time-to-first-token) — the delay before the first token appears. This is what a user feels in a streaming UI. It’s dominated by prefill: processing the prompt before generation starts, which for Anvil means the tool schemas plus the user message. Long prompts and big tool lists push TTFT up.
  • Total latency — time until the complete response (the whole tool-call JSON here). What an automated caller waits for when it isn’t streaming.
  • Throughput (tokens/s, requests/s) — only really meaningful under concurrency / batching. A single-request number is a sanity check, not a throughput measurement; real throughput under load is a Turbine subject.

The lab measures total latency over repeated calls and reports mean / p50 / p95 — honest percentiles beat a single lucky number:

def benchmark_latency(base_url: str, request: dict, *, n: int = 20) -> dict:
    """Measure latency over n calls against a running endpoint."""
    import time
    lat = []
    for _ in range(n):
        t0 = time.perf_counter()
        call_endpoint(base_url, request)
        lat.append(time.perf_counter() - t0)
    lat.sort()
    return {"n": n, "mean_s": sum(lat) / n, "p50_s": lat[n // 2], "p95_s": lat[int(n * 0.95)]}

Why p50 and p95? The mean hides tail behavior; p95 is the slow request your users actually notice. To capture TTFT specifically you’d stream the response and stop the clock at the first content chunk — the lab leaves that as a “Try it yourself” extension on top of benchmark_latency.

Treat vLLM (GPU) and Ollama (CPU) as two regimes, not a race. On a T4, vLLM will return a tool call in a fraction of a second; on a laptop CPU, Ollama is slower but costs $0 and needs no infrastructure. Present whatever you measure as “on my run, June 2026,” never as a constant — your hardware, prompt length, and quant type all move the number.

vLLM / Ollama / TGI at a glance

vLLMOllamaTGI
Serves the adapter or the merged model?the adapter, no merge (--lora-modules)the merged GGUFmultiple adapters on one base
HardwareNVIDIA GPU (CUDA)CPU / Apple Silicon (GPU optional)NVIDIA GPU
Multi-LoRA?yes — per request, by model nameno — one model per Modelfileyes — LORA_ADAPTERS, at scale
When to use ita GPU endpoint, especially multi-adapterlocal / CPU / desktop / air-gapped, zero inframulti-tenant multi-LoRA at scale

The one-line rule: adapter → vLLM or TGI; merged GGUF → Ollama.

⚠️ Common misconception (two linked halves).

(a) “vLLM serving the adapter and Ollama serving the merged GGUF are interchangeable — pick whichever.” No. They are different artifacts for different targets. vLLM applies the adapter on top of the base held in GPU memory (multi-LoRA, no merge). Ollama loads an already-merged GGUF (CPU / Apple Silicon, standalone). You cannot serve a raw adapter as a GGUF — you must merge_and_unload first (Module 12) — and merging just to run on vLLM throws away the share-one-base multi-LoRA win. The artifact you hold dictates the server.

(b) “I can expose VLLM_ALLOW_RUNTIME_LORA_UPDATING to hot-load adapters in prod — it’s so convenient.” No — vLLM flags this as a security risk. The /v1/load_lora_adapter endpoint lets a caller load arbitrary weights from an arbitrary path/URL into your running server. Keep it to internal dev/CI; never expose it on an untrusted or public endpoint. As of vLLM 0.23.0, June 2026.

Build it: serve Anvil two ways and measure it

Goal: serve Anvil with vLLM over the LoRA adapter (--enable-lora) and with Ollama over the merged GGUF (M12), call each with the same tool-call prompt, confirm the tool-call JSON is valid, and measure latency.

What you’ll see: the rendered vllm serve command and Ollama Modelfile; the validated tool-call JSON returned by each endpoint; and a latency report ({"n": 20, "mean_s": ..., "p50_s": ..., "p95_s": ...}) for vLLM (GPU) and Ollama (CPU) — two regimes, not a race.

The full, tested code lives in finetune-prod-labs/module-13. Its smoke test runs offline — no GPU, no network, no running server — so you can verify command construction and payload shape before you spin anything up.

Step 1 — Set up. Open the notebook on Colab/Kaggle (T4 runtime) for the vLLM path. Install the pinned stack including vllm==0.23.0; install Ollama (the local binary) for the CPU path; set HF_TOKEN if you pull the base from the Hub. Bring forward the inherited artifacts: the adapter (M4/M5/M10, e.g. outputs/anvil-dpo) and the merged GGUF (M12, ./anvil-q4_k_m.gguf). Nothing is re-trained, re-merged, or re-quantized here — you serve what you already built.

Step 2 — Serve the adapter via vLLM. Render the command and run it:

from anvil import serve
cmd = serve.vllm_serve_command("Qwen/Qwen3-4B-Base", adapter_path="outputs/anvil-dpo")
print(" ".join(cmd))
# vllm serve Qwen/Qwen3-4B-Base --port 8000 --enable-lora
#   --lora-modules anvil=outputs/anvil-dpo --max-lora-rank 16

Launch it in a shell, then wait for the endpoint to come up (poll GET /v1/models until it lists anvil).

Step 3 — Serve the merged GGUF via Ollama. Write the Modelfile and register it:

print(serve.ollama_modelfile("./anvil-q4_k_m.gguf"))
# FROM ./anvil-q4_k_m.gguf
# PARAMETER stop "<|im_end|>"
# PARAMETER temperature 0

Save that as Modelfile, then ollama create anvil -f Modelfile. Ollama now serves on :11434.

Step 4 — Call both endpoints with the same prompt. Build one OpenAI-shaped tool-call request and POST it to each:

TOOLS = [{"type": "function", "function": {"name": "get_weather",
          "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}]
req = serve.build_tool_call_request("What's the weather in Paris?", TOOLS)

vllm_resp = serve.call_endpoint("http://localhost:8000", req)      # GPU, adapter
ollama_resp = serve.call_endpoint("http://localhost:11434", req)   # CPU, merged GGUF

Confirm each response is a valid tool call with the right name using Anvil’s Module-7 parser (eval.parse_tool_calls) — a serving smoke check, not the formal eval.

Step 5 — Measure latency. Run serve.benchmark_latency against each endpoint and compare in prose (GPU vs CPU = two regimes):

print(serve.benchmark_latency("http://localhost:8000", req, n=20))   # vLLM / T4
print(serve.benchmark_latency("http://localhost:11434", req, n=20))  # Ollama / CPU

Step 6 — Cost/time. Record the wall-clock and write it into the frozen cost line at the end of lab.md. Both paths are free here (free T4 for vLLM, local CPU for Ollama).

Step 7 — The offline smoke test. The suite mocks the server: it asserts that serve.py builds the right vllm serve command, the right Modelfile, and the right LORA_ADAPTERS env, and that the tool-call payload is OpenAI-shaped — all without launching vLLM or Ollama. The real server calls (call_endpoint, benchmark_latency) need a running server, so they’re exercised only in the notebook, not in the offline suite (it never hits a live endpoint).

def test_vllm_command_serves_adapter_when_given():
    base = serve.vllm_serve_command("Qwen/Qwen3-4B-Base")
    assert "--enable-lora" not in base                 # no adapter -> serve base directly
    with_adapter = serve.vllm_serve_command("Qwen/Qwen3-4B-Base", adapter_path="anvil-sft")
    assert "--enable-lora" in with_adapter
    assert "anvil=anvil-sft" in with_adapter

def test_ollama_modelfile_points_at_gguf():
    mf = serve.ollama_modelfile("./anvil-q4_k_m.gguf")
    assert mf.startswith("FROM ./anvil-q4_k_m.gguf")
    assert "<|im_end|>" in mf                           # the Qwen3 stop token

Run it from the repo root:

uv run pytest module-13/tests/
# 4 passed

Try it yourself:

  1. Load two adapters on vLLM — pass --lora-modules anvil=outputs/anvil-dpo other=outputs/anvil-sft — and route by request (model="anvil" vs model="other"). Watch multi-LoRA in action: one base in VRAM, two specialists.
  2. Add a TTFT measurement. Extend benchmark_latency to stream the response and stop the clock at the first content chunk, then compare TTFT vs total latency on vLLM (GPU) and Ollama (CPU) for the same prompt — quantify the “GPU throughput vs zero-infra CPU” trade-off.

What this lab does NOT do. No re-quantization or re-merge (M12 — you reuse the existing GGUF and adapter). No exposing VLLM_ALLOW_RUNTIME_LORA_UPDATING (explained as a risk, never enabled on an exposed endpoint). No full TGI deployment (multi-LoRA is surveyed and the LORA_ADAPTERS env is built, not deployed). No scaling / batching / autoscaling / KV-cache tuning — that’s Turbine; here you measure single-request latency, not throughput under load. No push_to_hub or model card (M14 — you serve locally, you don’t publish).

In production

Outside the lab, the first thing that changes is the mindset: a model.generate() loop is not a service. A service has an HTTP endpoint, stays warm in memory, handles concurrency, and exposes metrics — so in production you put a server in front of the model (vLLM or TGI on a GPU, Ollama for local/edge), not a Python loop.

The choose-by-artifact decision becomes an architecture decision. If you serve many specialists, keep the adapters and do multi-LoRA — vLLM or TGI hold one base in memory and route N adapters by request, so you pay for roughly one model, not N. If you want a self-contained binary for a desktop app, a CPU box, the edge, or an air-gapped environment, merge once and serve the GGUF via Ollama. The personal rule I follow: I serve the adapter on vLLM for the GPU endpoint and keep a merged GGUF in Ollama for local smoke-testing — same model, two regimes.

On security: never expose runtime LoRA loading (VLLM_ALLOW_RUNTIME_LORA_UPDATING + /v1/load_lora_adapter) on a public endpoint — vLLM itself flags it as an arbitrary-load vector. Keep it to internal dev/CI behind your network boundary.

On cost (decision F6): a GPU serving endpoint runs continuously. This isn’t ”~$2 for one training run” anymore — it’s a standing cost, $/hour × uptime, whether or not anyone is calling it. Before you size that endpoint, measure real throughput under load, not single-request latency — and the moment you’re tuning batching, KV-cache, and autoscaling to drive cost-per-request down, you’ve crossed into the Turbine course. This module hands you the foundations: which server, which artifact, how to call it, how to read a latency number honestly.

Concept check

Why this matters. This module owns the serving half of T11. The skills it anchors: a model on disk is not an endpoint (you need a server); vLLM serves the adapter (--enable-lora / --lora-modules, base loaded once, routed per request, no merge) while Ollama serves the merged GGUF (Modelfile FROM ./anvil.gguf, local REST on :11434, CPU / Apple Silicon); TGI does multi-LoRA at scale (LORA_ADAPTERS); multi-LoRA means deploy the base once and serve N adapters (keep adapters, merge only for a standalone binary); you measure latency (mean/p50/p95, plus TTFT) instead of guessing; and runtime-LoRA loading is a security risk vLLM warns about — never exposed. serve.py is the last technical link before publishing (M14), and the intro to the future Turbine course (scaling).

  1. You have a trained LoRA adapter for Anvil and want a GPU endpoint without merging anything. Which server, and why not the other obvious one?

    • A. Ollama, because it serves any fine-tuned model.
    • B. vLLM with --enable-lora / --lora-modules, targeting the adapter by model name per request; Ollama is wrong because it serves a merged GGUF, not a raw adapter.
    • C. TGI, because it’s the only server that supports adapters.
    • D. A model.generate() loop behind Flask — simplest.

  2. You need to serve Anvil on a laptop with no NVIDIA GPU. What do you serve, and how?

    • A. The raw LoRA adapter through Ollama.
    • B. The merged GGUF through Ollama (Modelfile FROM ./anvil.gguf, API on :11434) — CPU / Apple Silicon.
    • C. vLLM with --enable-lora on the CPU.
    • D. TGI with LORA_ADAPTERS on the CPU.

  3. A teammate wants to set VLLM_ALLOW_RUNTIME_LORA_UPDATING=1 on the public production endpoint so they can hot-load new adapters. What do you tell them?

    • A. Go for it — it’s a documented convenience feature.
    • B. It’s fine as long as the adapters are small.
    • C. Don’t — vLLM flags it as a security risk: /v1/load_lora_adapter lets a caller load arbitrary weights from an arbitrary path/URL. Keep it to internal dev/CI, never on a public endpoint.
    • D. It only works on Ollama anyway.

  4. You need to serve 10 Anvil specialists without paying for 10 models. What’s the move?

    • A. Merge each specialist and run 10 Ollama instances.
    • B. Keep the 10 adapters and do multi-LoRA — vLLM or TGI load the base once and route each request to the right adapter by name; you pay for ~one model plus 10 small adapters.
    • C. Merge all 10 into one GGUF.
    • D. Run 10 separate vLLM servers, one per merged model.

  5. On a streaming endpoint you see a high TTFT but a decent overall token rate. What does TTFT measure, and why isn’t real throughput captured by this single-request test?

    • A. TTFT is total latency; throughput is captured fine here.
    • B. TTFT is the time to the first token — dominated by prefill (prompt length + tool schemas); a single-request latency number is a sanity check, and real throughput under concurrency/batching is a Turbine subject, not measured here.
    • C. TTFT is the number of tokens; throughput needs no concurrency to measure.
    • D. TTFT measures GPU memory; throughput is irrelevant for tool calls.

Answers.

  1. B. vLLM serves a LoRA adapter directly with --enable-lora + --lora-modules anvil=<path>, and you target it per request with model="anvil" — no merge. Ollama serves a merged GGUF, so it can’t take a raw adapter (A is wrong). TGI also serves adapters, but it isn’t the only one (C is wrong), and a Flask-wrapped loop is a demo, not a service (D).
  2. B. No GPU means the Ollama / GGUF path: serve the merged GGUF (Module 12) via a Modelfile FROM ./anvil.gguf on :11434, on CPU / Apple Silicon. A is impossible — you can’t serve a raw adapter as a GGUF (merge first). vLLM and TGI both need a CUDA GPU (C, D).
  3. C. vLLM itself warns that runtime LoRA loading is a security risk: the /v1/load_lora_adapter endpoint loads arbitrary weights from an arbitrary path/URL into the running server. It’s fine for internal dev/CI behind your boundary, but never on an untrusted/public endpoint. Adapter size (B) is irrelevant; it’s a vLLM feature, not Ollama-only (D).
  4. B. Keep the adapters and do multi-LoRA: vLLM (per-request) or TGI (LORA_ADAPTERS) load one base into memory and route by adapter name, so 10 specialists cost ~one model plus 10 small adapters. Merging each (A, C, D) throws away exactly that share-one-base economics.
  5. B. TTFT is time-to-first-token, dominated by prefill — processing the prompt and tool schemas before generation starts. A single-request latency number is a sanity check; real throughput only appears under concurrency/batching, which this module deliberately doesn’t measure (that’s Turbine). A, C, and D misdefine TTFT.

Common pitfalls.

  • Serving a raw LoRA adapter as a GGUF. Impossible — a GGUF is a merged model; you must merge_and_unload first (Module 12; the llama.cpp converter needs full weights). The inverse trap: merging just to run on vLLM, which throws away the multi-LoRA “one base, many adapters” win.
  • Exposing runtime LoRA loading. VLLM_ALLOW_RUNTIME_LORA_UPDATING + /v1/load_lora_adapter is a security risk vLLM flags itself — arbitrary weight loading. Internal dev/CI only, never a public endpoint. As of vLLM 0.23.0, June 2026.
  • Confusing model.generate() with a service. A Python loop is not an endpoint: no HTTP API, no warm model, no concurrency, no metrics. A server (vLLM / Ollama / TGI) is. And don’t conflate serving with scaling — continuous batching, KV-cache, and autoscaling are the Turbine course, not this one.

Key takeaways

  • A model on disk is not an endpoint. A model.generate() loop is a demo; a service needs a server (vLLM / Ollama / TGI) with an HTTP API, a warm model, concurrency, and metrics.
  • vLLM serves the adapter — --enable-lora + --lora-modules anvil=<path>, base loaded once, routed per request by model name, no merge. Set --max-lora-rank ≥ your adapter’s rank.
  • Ollama serves the merged GGUF — Modelfile FROM ./anvil.gguf, local REST on :11434, CPU / Apple Silicon, no GPU. You cannot serve a raw adapter this way (merge first, M12).
  • TGI does multi-LoRA at scale (LORA_ADAPTERS); the lever both vLLM and TGI give you is deploy the base once, serve N adapters — keep adapters, merge only for a standalone binary.
  • Measure, don’t guess — mean/p50/p95 latency over repeated calls, with TTFT (dominated by prefill) as the metric a streaming user feels. vLLM (GPU) and Ollama (CPU) are two regimes, not a race.
  • Runtime LoRA loading is a security risk vLLM warns about — VLLM_ALLOW_RUNTIME_LORA_UPDATING / /v1/load_lora_adapter loads arbitrary weights; never expose it. As of vLLM 0.23.0, June 2026.
  • Scaling is a different subject. Continuous batching, paged KV-cache, speculative decoding, autoscaling — all of it is the future Turbine course. This module is the foundations.

What’s next

Anvil is served locally now — time to make it public. Module 14 publishes Anvil to the Hugging Face Hub with a complete model card (base_model: Qwen/Qwen3-4B-Base, the base-vs-fine-tuned evals from Module 7) so anyone can pull and serve it. That’s your portfolio artifact — the published model that stands in for a certificate.

Want the full fine-tuning concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · ← Module 12: Shrink It for Serving · Module 14: Ship It — Publishing, Model Cards, and Reproducibility → · Lab code for this module

References