Reliable Agents: Structured Output, Validation, and Errors (smolagents, Module 8)

This is Module 8 of smolagents Mastery, a free 15-module course that takes you from your first code agent to a production-grade one. Start at Module 1 or browse the full syllabus.

The Quill you built in Module 7 is smart and frugal. It plans, it profiles a dataset once, it stops re-loading the same CSV. On a good day, it answers your question in four steps and a chart pops out. That is the day you remember.

Here is the day you forget. Run the same question three times. The first run hands you a tidy paragraph. The second hands you a half-built Python dict you have to parse by eye. The third says “growth is strong” — no number, no chart saved to disk, and a confident sentence about “the public trend” with no URL anywhere. Three runs, three shapes, and not one of them is something you could audit, diff against last week, or hand to a colleague. Quill looks like it works. It is not reliable.

That gap matters more than it seems. The moment you wire up automated evaluation (Module 14) or cited RAG (Module 12), this fuzziness makes Quill useless — you cannot score a report whose shape changes every run, and you cannot cite sources that aren’t there. This module closes the gap. You turn Quill’s answer into a contract: a structured, validated QuillReport that refuses to finish without a saved chart or a source for any web-backed claim — and self-corrects when a check rejects it.

In this module

You’ll learn

Distinguish smolagents’ two structured-output mechanisms — response_format (model level) vs use_structured_outputs_internally (a CodeAgent argument) — and know which works with which backend.

Validate a final answer with final_answer_checks (3-arg validators (final_answer, memory, agent)) and understand why a False loops the agent instead of crashing.

Map the full AgentError hierarchy and explain the self-correction loop — the error lands in ActionStep.error, goes back to the LLM, and the model fixes it on the next step.

Diagnose what makes an agent reliable in production: where validation belongs, and when structured outputs help (and when they validate nothing).

You’ll build — Quill now returns a validated QuillReport, and refuses to finish without a saved chart or a source for any web-backed claim, self-correcting after a failed check.

Theory areas covered

T8 — Reliability: structured output, validation, error handling — 6% of the course. This module owns it.

Prerequisites — Module 2 (final_answer, RunResult, the intro to AgentError), Module 3 (save_chart and the web tools), Module 4 (make_model(), the Model classes, structured-output providers), and Module 7 (the CodeAgent system prompt + default YAML). HF_TOKEN configured in your .env (Module 1).

Where you are in the series

✅ Modules 1–7 — your first CodeAgent, the ReAct loop, Quill’s data toolbox, make_model(), the sandbox, memory + callbacks, planning + sharpened instructions.
👉 Module 8 — Reliable agents. You are here.
⬜ Modules 9–15 — MCP and the Hub, multi-agents, multimodal, RAG, deployment, telemetry, and the capstone.

Quill before this module: it returns a free-form answer — raw text, sometimes a chart path, with no guarantee of structure, no guarantee a chart was actually saved, and no guarantee a web claim is cited. Quill after this module: it returns a validated QuillReport, refuses to conclude without a saved chart or a source for any web-backed claim, and you watch it self-correct after a rejected report.

The contract you freeze here is not a one-module convenience. The QuillReport schema is reused by Module 12 (which cites its sources) and Module 14 (which scores its quality). Get it right now; freeze it; build on it.

Two kinds of structured output (and the trap of conflating them)

Type “smolagents structured output” into a search box and you get two completely different answers, often in the same blog post, never distinguished. That conflation is the single most common mistake in this area, so let’s name the two mechanisms before we use either.

Mechanism A — response_format (model level). This lives on the model, not the agent. The signature, as of smolagents 1.26.0, is:

Model.generate(messages, stop_sequences=None, response_format=None, tools_to_call_from=None, **kwargs)

response_format is a JSON schema you hand the model so its next reply comes back as JSON matching that schema. What it actually does depends entirely on the backend — and this is the “why” worth learning:

OpenAIModel / AzureOpenAIModel — the schema is forwarded to the OpenAI API through _prepare_completion_kwargs(). Full JSON-schema support.
LiteLLMModel — forwarded to litellm.completion(); support depends on the underlying provider LiteLLM is routing to.
InferenceClientModel — restricted. As of smolagents 1.26.0, the source guards it:
```
STRUCTURED_GENERATION_PROVIDERS = ["cerebras", "fireworks-ai"]
if response_format is not None and self.client_kwargs["provider"] not in STRUCTURED_GENERATION_PROVIDERS:
    raise ValueError("InferenceClientModel only supports structured outputs with these providers: ...")
```
So with InferenceClientModel, a response_format only works when provider="cerebras" or provider="fireworks-ai". Anywhere else, you get a ValueError. That list is narrow today and will almost certainly grow — treat it as “as of smolagents 1.26.0” and check the source or docs for the current set, never hard-code an assumption that it is fixed.

Mechanism B — use_structured_outputs_internally (a CodeAgent argument, not a model’s). This is a bool, default False, on CodeAgent.__init__ — it is not a model setting, and that distinction trips people constantly. When you set it to True, two things happen under the hood (source-level behavior, smolagents 1.26.0):

CodeAgent loads a different default prompt YAML — structured_code_agent.yaml instead of the usual code_agent.yaml.
Inside its step loop, it sets response_format on every model call to a built-in schema (CODEAGENT_RESPONSE_FORMAT) so the model returns its Thought + code as a structured object each step.

The point of Mechanism B is parsing reliability: it makes each step’s output easier to parse correctly. It is not “validate the final answer.” It never looks at the meaning of your answer — only the shape of each step’s reply.

Here is the cross-consequence to burn into memory: because Mechanism B drives response_format under the hood, use_structured_outputs_internally=True with an InferenceClientModel only works on those same two whitelisted providers. Turn it on with the default HF backend and any other provider, and you inherit Mechanism A’s ValueError. That is exactly why Quill routes every model through make_model() (Module 4) — one place to swap to a response_format-capable backend if you ever want this.

One more thing you will see in old tutorials: a grammar= argument on generate. grammar= was removed in smolagents 1.21.0. It used to drive constrained generation (a TGI-flavored mechanism); the modern path is response_format=. If you see grammar= in a snippet, the snippet is stale. We will never write it as live code.

`response_format` vs `use_structured_outputs_internally`

	`response_format`	`use_structured_outputs_internally`
Lives on	`Model.generate(...)` (a model)	`CodeAgent.__init__(...)` (the agent)
What it controls	the JSON shape of one model call’s reply	the Thought+code shape of every step
Backend constraint	`InferenceClientModel` → only `cerebras` / `fireworks-ai`	inherits the same constraint (it drives `response_format`)
YAML loaded	—	`structured_code_agent.yaml` (instead of `code_agent.yaml`)
Validates the final answer’s content?	No	No

That last row is the headline. Neither mechanism validates your answer. They guarantee form, never meaning. Validating the meaning — “there is a chart,” “there is a source” — is a different job, done by a different tool, at a different point in the loop. That tool is final_answer_checks, and where it sits is the whole point of the next diagram.

flowchart TD
    subgraph loop["CodeAgent step loop (runs every step)"]
        gen["Model.generate(...)"]
        rf["response_format<br/>(shapes this call's reply)"]
        yaml["use_structured_outputs_internally=True<br/>→ loads structured_code_agent.yaml<br/>→ sets response_format every step"]
        exec["execute the parsed code"]
        rf -.->|drives| gen
        yaml -.->|drives| gen
        gen --> exec
        exec -->|not final answer| gen
    end
    exec -->|final answer detected| checks{"final_answer_checks<br/>(validate CONTENT)"}
    checks -->|pass| done["accept → return QuillReport"]
    checks -->|reject| loop

Structured outputs act inside the loop (they shape what the model emits each step). Validation acts at the exit (it inspects the finished answer and can bounce it back in). Two mechanisms, two jobs, two places. (Owns T8.4, T8.5, T8.6; reinforces T7.3’s structured_code_agent.yaml and T4.)

`final_answer_checks`: turning the answer into a contract

Recall how a run ends (Module 2). The agent terminates by calling the FinalAnswerTool — name="final_answer", output_type="any", and its forward() returns its argument unchanged. So agent.run(...) hands you back whatever the model passed to final_answer(...); pass return_full_result=True and you get a RunResult instead (output, state, steps, token_usage, timing). Until now, that payload could be anything. We are about to make it a contract.

final_answer_checks is an argument of MultiStepAgent.__init__ (list[Callable] | None). Each callable is a validator that runs on the final answer before the agent accepts it. Two details decide whether you use it correctly.

Detail 1 — the signature is 3 arguments. A check is def check(final_answer, memory, agent) -> bool. This is the part most tutorials get wrong. The smolagents 1.26.0 call site, in agents.py’s _validate_final_answer, is exact:

def _validate_final_answer(self, final_answer):
    for check_function in self.final_answer_checks:
        try:
            assert check_function(final_answer, self.memory, agent=self)
        except Exception as e:
            raise AgentError(f"Check {check_function.__name__} failed with error: {e}", self.logger)

The 2-arg example you’ve seen in the guided tour (is_integer(final_answer, agent_memory=None)) only “works” because the third argument is passed as a keyword (agent=self) — Python lets it bind to a parameter with a default. That is luck, not design. Write the robust 3-arg form: (final_answer, memory, agent).

Detail 2 — what False means. Look at that call site again. The check runs inside a try, and the result feeds an assert. Two ways to fail it: return False (the assert fails) or raise an exception (caught directly). Either way, smolagents wraps it in an AgentError — and an AgentError is caught by the run loop, not propagated. It lands in ActionStep.error and goes back to the model. The run does not crash; it loops. Your validation becomes a self-correction signal: Quill re-reads why its report was rejected and fixes it on the next step.

Because the model only reads the failure message, the message is your interface. Returning a bare False gives the model nothing to act on (the wrapped error just says the check “failed”). Raising ValueError("...") with an actionable message tells it exactly what to do. So the recommended pattern is to raise with a fix-it sentence — and that is exactly what Quill’s checks do:

def check_has_chart(final_answer, memory, agent) -> bool:
    """3-arg final_answer_check: a report must include at least one saved chart."""
    if not isinstance(final_answer, QuillReport):
        raise ValueError(
            "Final answer rejected: the answer must be a QuillReport "
            "(import it from quill.report and pass it to final_answer), not "
            f"a {type(final_answer).__name__}."
        )
    if not final_answer.chart_paths:
        raise ValueError(
            "Final answer rejected: a report must include at least one saved chart. "
            "Draw a matplotlib chart, call save_chart to save it, add the returned path to "
            "QuillReport.chart_paths, then call final_answer again."
        )
    return True

This check runs after smolagents detects the final answer (is_final_answer) and before it accepts it. A non-QuillReport? Rejected — wrong shape. A QuillReport with chart_paths=[]? Rejected — wrong content. Both bounce back as a captured AgentError, and the model self-corrects.

⚠️ Common misconception: “Structured output validates my answer.”

It does not. response_format and use_structured_outputs_internally guarantee a form — well-typed JSON, a parseable Thought+code block. They never guarantee correct content. A perfectly-typed QuillReport can still have chart_paths=[] and sources=[] — it is valid JSON and a worthless report. Only a final_answer_check enforces the business rule (“there is at least one source when there are web claims”). Form is not validity. Keep the two jobs separate, and never reach for structured outputs to do a validator’s work.

(Owns T8.3; reinforces T8.1 FinalAnswerTool, T8.2 RunResult, T1.12 is_final_answer/_validate_final_answer.)

The `AgentError` hierarchy and the self-correction loop

You just saw a rejected check become an AgentError that the loop catches. To reason about reliability you need the whole error family, because each member tells you a different thing about where a run went wrong. Here is the complete hierarchy as of smolagents 1.26.0 (utils.py), verified against the installed source:

Class	Parent	What it means
`AgentError`	`Exception`	base class; logs through the agent’s logger
`AgentParsingError`	`AgentError`	the model’s output couldn’t be parsed (malformed code/JSON)
`AgentExecutionError`	`AgentError`	something failed during the execution phase
`AgentMaxStepsError`	`AgentError`	the step budget ran out
`AgentGenerationError`	`AgentError`	the model’s generation call itself errored
`AgentToolCallError`	`AgentExecutionError`	the LLM passed bad arguments to a tool
`AgentToolExecutionError`	`AgentExecutionError`	the tool itself raised at runtime

flowchart TD
    AE[AgentError] --> APE[AgentParsingError]
    AE --> AEE[AgentExecutionError]
    AE --> AMSE[AgentMaxStepsError]
    AE --> AGE[AgentGenerationError]
    AEE --> ATCE[AgentToolCallError]
    AEE --> ATEE[AgentToolExecutionError]

Note the shape: AgentToolCallError and AgentToolExecutionError are not direct children of AgentError — they descend from AgentExecutionError. That matters when you except on these: catching AgentExecutionError catches both tool failures; catching AgentError catches everything. The two tool errors split a useful distinction — bad arguments from the LLM (AgentToolCallError) versus the tool blew up inside (AgentToolExecutionError). The first is the model’s fault; the second is your tool’s.

Why none of this crashes your run. When an error occurs in a step, smolagents captures it in ActionStep.error (type AgentError | None) instead of letting it propagate. The error is then re-injected into the model’s context: write_memory_to_messages() serializes each step back into the conversation, and a step carrying an error adds that message — so on the next turn the model sees what went wrong and can fix it. The docs put it plainly: “most of the errors are just ‘LLM dumb’ kind of errors, from which the LLM auto-corrects in the next step.”

There is exactly one error that ends a run: AgentMaxStepsError. When the step budget runs out, the run stops with state="max_steps_error". Everything else is fuel for the next step.

Now connect the two halves of this module: a final_answer_check that returns False uses this exact mechanism. The wrapped AgentError lands in ActionStep.error, gets re-fed to the model, and the model self-corrects — the same recovery path as a KeyError in the agent’s own pandas code. That is why an actionable check message is so valuable: it is the one thing the model reads to understand the rejection.

The parsing layer underneath. None of this would be robust without the helpers in utils.py that turn noisy LLM text into something executable. parse_code_blobs() extracts the code from a model’s reply (with a markdown fallback when the model wraps it loosely); parse_json_blob() does the same for JSON; truncate_content() caps observations (default max_length=20000 characters, verified in 1.26.0) so a giant print() cannot blow up the context window. These exist because LLMs emit prose around the code you actually want — the parsing layer tolerates that noise and bounds the size of what comes back. It is the quiet machinery that makes an imperfect model output a runnable action. (Owns T8.7, T8.8, T8.9; reinforces T1.13 max_steps, T6.5 write_memory_to_messages.)

Designing a reliable report contract for Quill

Now we assemble the contract. The schema is frozen here — this is the module that introduces it, and Module 12 (citations) and Module 14 (eval) both depend on the exact shape. Adding a sixth field later would silently break those modules, so the schema is a contract you do not casually edit.

from dataclasses import dataclass, field

@dataclass
class Source:
    url: str
    title: str

@dataclass
class QuillReport:
    question: str
    findings: list[str] = field(default_factory=list)
    chart_paths: list[str] = field(default_factory=list)
    sources: list[Source] = field(default_factory=list)
    caveats: list[str] = field(default_factory=list)

Five fields, no more. A QuillReport carries the restated question, the findings (short standalone claims, each able to carry a [n] citation), the chart_paths (evidence save_chart produced), the sources (the cited Source objects), and honest caveats. A to_markdown() method renders it: each finding keeps its [n] markers, and a Sources section lists [n] [title](url) so the numbers line up. A one-source report contains the marker [1] in the body and [1] [<title>](<url>) in the Sources section — that rendered Markdown is the artefact Module 12 cites against and Module 14 scores. We freeze it now precisely because downstream modules build on it.

The second check is conditional. A purely local analysis needs no sources; a web-backed one must cite at least one. So the check first asks “did this run actually go to the web?” and only then demands a source:

WEB_TOOL_NAMES = ("web_search", "visit_webpage")  # WebSearchTool().name, VisitWebpageTool().name

def _used_a_web_tool(memory) -> bool:
    """Did this run call a web tool? Scan each step's code (the supported way to read a run)."""
    steps = getattr(memory, "steps", None) or []
    for step in steps:
        code = getattr(step, "code_action", None) or ""
        for call in getattr(step, "tool_calls", None) or []:
            args = getattr(call, "arguments", None)
            if isinstance(args, str):
                code += "\n" + args
        if any(name in code for name in WEB_TOOL_NAMES):
            return True
    return False

def check_has_source_for_web_claims(final_answer, memory, agent) -> bool:
    """3-arg final_answer_check: a web-backed report must cite at least one source."""
    if not isinstance(final_answer, QuillReport):
        raise ValueError("Final answer rejected: the answer must be a QuillReport, not "
                         f"a {type(final_answer).__name__}.")
    if _used_a_web_tool(memory) and not final_answer.sources:
        raise ValueError(
            "Final answer rejected: this analysis used the web (web_search/visit_webpage) but "
            "QuillReport.sources is empty. Add a Source(url=..., title=...) for every web-backed "
            "claim, cite it as [n] in the matching finding, then call final_answer again."
        )
    return True

Two design notes you should steal. First, _used_a_web_tool reads memory.steps — the supported way to inspect a run. It never touches agent.logs, which was removed in smolagents 1.21.0; if a snippet reads .logs, it is stale. Second, there is a real timing subtlety: the validator runs before smolagents appends the current step to memory, so a web call made in the same step as final_answer is not yet visible. In practice Quill searches the web in earlier exploration steps and answers later, so the evidence is already in memory by the time the check reads it — and a web call in the final step would simply be caught on the next attempt.

Where reliability lives. Be deliberate about which mechanism owns which guarantee — this is the engineering decision the whole module builds toward:

Concern	Where it lives in Quill
Output shape (it is a `QuillReport`)	the Python code Quill writes (optionally `use_structured_outputs_internally`)
Business rules (chart present; source for web claims)	`final_answer_checks`
Transient failures (a `KeyError`, a bad column)	self-correction via `ActionStep.error` + a retried step
Hard stop (a loop that never satisfies a check)	`max_steps` → `state="max_steps_error"`

The form of the report is guaranteed by the code Quill writes (a QuillReport dataclass it constructs in the sandbox). The business rules live in final_answer_checks. We deliberately do not lean on response_format for either — partly because it is the wrong tool for content validation, and partly because Quill’s default InferenceClientModel does not even support it outside cerebras/fireworks-ai. One mechanism per job. (Applies T8.3 + the frozen schema; reinforces T8.1/T8.2.)

Build it

Goal: turn Quill’s free-form answer into a contract. Quill returns a QuillReport, and final_answer_checks refuse it if there is no saved chart, or a web-backed claim with no source — then you watch it self-correct. The full, tested code is at smolagents-course-labs/module-08; every snippet here is drawn from code that passes pytest offline.

1. Setup. Carry the Module 7 state forward (the cumulative rule: M8 code must still pass the M1–M7 smoke tests), then sync the pins. No new extra — final_answer_checks is core smolagents and QuillReport is a plain dataclass.

uv venv --python 3.11
uv pip install "smolagents[toolkit,litellm,openai,docker]==1.26.0" \
  "huggingface_hub>=1.0,<2" "pandas>=2.2.3" matplotlib
cp module-08/.env.example module-08/.env   # then add your HF token; never commit .env

2. Freeze the schema and write the checks in quill/report.py — the QuillReport/Source dataclasses, to_markdown(), and the two 3-arg checks shown above. The list build_quill wires in is just:

QUILL_FINAL_ANSWER_CHECKS = [check_has_chart, check_has_source_for_web_claims]

def quill_final_answer_checks() -> list:
    """Return a FRESH copy so a caller cannot mutate ours."""
    return list(QUILL_FINAL_ANSWER_CHECKS)

3. Wire the checks into build_quill. The signature is extended by addition — every prior call site still works. Omitting final_answer_checks gives Quill’s defaults; [] opts out (the bench’s bare baseline):

agent = CodeAgent(
    tools=[load_dataset, profile_dataframe, save_chart(),
           WebSearchTool(), VisitWebpageTool()],
    model=model or make_model(role="analyst"),   # M4: one place to swap the backend
    executor_type=executor_type,                  # M5: from resolve_executor()
    additional_authorized_imports=authorized_imports,
    step_callbacks=quill_callbacks(),             # M6
    planning_interval=planning_interval,          # M7
    instructions=resolved_instructions,           # M7
    final_answer_checks=resolved_checks,          # M8: validate CONTENT
    use_structured_outputs_internally=use_structured_outputs_internally,  # M8: OFF by default
    max_steps=8,
)

Keep the model as make_model() (Module 4). Do not turn on response_format for the default InferenceClientModel — it only supports it with provider in {"cerebras","fireworks-ai"} as of smolagents 1.26.0. There is one more wiring detail: build_quill injects QuillReport/Source into the sandbox via the executor’s send_variables hook, so the agent’s generated code can build a report without import quill.report — the frozen least-privilege import lock (Module 5) deliberately forbids that import, and we never widen it.

4. Run it. The canonical command asks Quill for a validated report:

uv run python -m quill "Which category grew fastest, and is that consistent with the public trend?" --data data/sales.csv

[Quill] Backend: hf | Model: Qwen/Qwen2.5-Coder-32B-Instruct
...
Final answer rejected: a report must include at least one saved chart. Draw a matplotlib
chart, call save_chart to save it, add the returned path to QuillReport.chart_paths, then
call final_answer again.
...                       # ← Quill self-corrects: draws + saves a chart, answers again
===== REPORT =====
# Which category grew fastest, and is that consistent with the public trend?

## Findings
- Team grew fastest, +38% from Q1 to Q4 [1].

## Charts
- `outputs/category_growth.png`

## Sources
[1] [SaaS Trends 2025](https://example.com/saas-2025)
[Quill] Run cost — input tokens: 12,431 | output tokens: 1,118 | total: 13,549

The exact trajectory varies — LLMs are non-deterministic. What is guaranteed is the shape: a QuillReport the checks accepted (a saved chart; a source for any web-backed claim).

5. See the self-correction in code (offline). The smoke tests prove the loop with a FakeModel — zero tokens, no network. Step 1 answers with no chart (rejected); step 2 draws, saves, and returns a complete report (accepted):

def test_self_correction_after_a_rejected_report(fake_model):
    step1 = ("rep = QuillReport(question='Which category grew fastest?', "
             "findings=['Team grew fastest'], chart_paths=[])\n"  # NO chart -> rejected
             "final_answer(rep)")
    step2 = ("import matplotlib; matplotlib.use('Agg')\n"
             "import matplotlib.pyplot as plt\n"
             "plt.figure(); plt.bar(['Free','Pro','Team'],[1,2,3])\n"
             "path = save_chart('m8_self_correction')\n"
             "rep = QuillReport(question='Which category grew fastest?', "
             "findings=['Team grew fastest'], chart_paths=[path])\n"
             "final_answer(rep)")
    agent = build_quill(model=fake_model([step1, step2]))
    out = agent.run(build_task(CSV, "Which category grew fastest?"))

    assert isinstance(out, QuillReport)
    action_steps = [s for s in agent.memory.steps if isinstance(s, ActionStep)]
    assert action_steps[0].error is not None          # step 1 was rejected (captured, not crashed)
    assert isinstance(action_steps[0].error, AgentError)
    assert action_steps[-1].error is None             # the corrected answer was accepted

That is the whole module in one test: a rejection lands in ActionStep.error (an AgentError, not a crash), and the next step fixes it. A companion test, test_bare_string_answer_is_rejected_and_loops_not_crashes, confirms a bare string never gets accepted — it loops to state == "max_steps_error" with the rejection captured every attempt.

Run the suite:

uv run pytest module-08/tests/                     # offline (live/sandbox auto-skip)
QUILL_LIVE_TESTS=1 uv run pytest module-08/tests/  # also the real validated-report run

124 passed, 9 skipped

The 9 skipped are the live and sandbox tests; the offline 124 cover the frozen schema, to_markdown()’s [1] citation, both checks, the end-to-end fake-model report, and the self-correction loop — all with no token spend.

Try it yourself (not graded):

A third check. Refuse a report with findings=[] (an analysis with no conclusion) or more than N caveats (a report drowning in hedges). Keep the 3-arg signature and an actionable message.
Real structured outputs. Wire make_model(backend="litellm") on a provider that supports response_format, turn on use_structured_outputs_internally=True, and compare the per-step parsing success rate against the default. Note it still doesn’t validate content — the checks do.

In production

Validation is not free. Every rejected report costs at least one extra LLM call — the agent has to read the rejection, re-plan, and answer again. In a tight loop that adds up fast, so bound your retries. I cap max_steps low enough that an agent which can never satisfy a check fails cleanly instead of burning 20 calls on a broken loop — Quill ships max_steps=8 against a default of 20, and I would not run a single-CSV job much higher.

Two failure modes are worth designing against. First, over-strict validation can trap the agent in permanent failure — if no reachable answer passes your checks, the agent loops to max_steps_error every time. Always give it an escape: prefer degrading with an explicit caveat (“source unavailable; finding is provisional”) over a hard crash. Second, the check message is the only thing the model reads to recover — a vague message (“invalid report”) gives it nothing to act on. Make every message name the fix.

One freshness reality: STRUCTURED_GENERATION_PROVIDERS is just two providers as of smolagents 1.26.0. Do not build an architecture that depends on response_format over InferenceClientModel — route through make_model() so you can swap to a LiteLLMModel/OpenAIModel without touching the agent.

In Module 14, these same final_answer_checks become the seed of an automated eval that scores every Quill report.

Concept check

Why this matters. Reliability is the line between “it seemed to work” and an agent you can actually operate. A free-form agent can’t be audited (you can’t read a shape that changes every run), can’t be evaluated (Module 14 needs a stable contract to score), and can’t be cited (Module 12 needs sources to exist). The three skills here — knowing which structured-output mechanism does what, validating content with final_answer_checks, and reading the AgentError self-correction loop — are what turn a demo into something you’d put in front of a colleague.

Answer all five, then check yourself against the grouped answers below.

1. A team wants every step of their CodeAgent to return a cleanly parseable Thought + code block. Which mechanism delivers that?

A. final_answer_checks
B. use_structured_outputs_internally=True
C. grammar="..." on generate
D. a stricter max_steps

2. A developer sets response_format=<schema> on an InferenceClientModel with provider="together" and gets a ValueError. Why, and what are the right fixes?

A. The schema is malformed; rewrite the JSON schema.
B. together isn’t in STRUCTURED_GENERATION_PROVIDERS; switch to cerebras/fireworks-ai, or use OpenAIModel/LiteLLMModel.
C. response_format doesn’t exist in 1.26.0; use grammar=.
D. You must also set use_structured_outputs_internally=True for it to work.

3. A final_answer_check returns False. What does the agent do?

A. Raises immediately and the run crashes.
B. Accepts the answer anyway and logs a warning.
C. Captures an AgentError in ActionStep.error, re-feeds it to the model, and loops so the model self-corrects.
D. Raises AgentMaxStepsError and stops.

4. The LLM passes a wrongly-typed argument to one of Quill’s tools. Which error class is raised, and what happens to the run?

A. AgentToolExecutionError; the run crashes.
B. AgentParsingError; the answer is rejected.
C. AgentToolCallError (a subclass of AgentExecutionError); it’s captured in ActionStep.error and re-fed so the model self-corrects.
D. AgentGenerationError; the model is retried with a new prompt.

5. Quill returns a perfectly-typed QuillReport with chart_paths=[] and sources=[] after a web search. Which mechanism would have caught this?

A. response_format — it enforces required fields.
B. use_structured_outputs_internally — it validates each step.
C. final_answer_checks — structured outputs guarantee form, not content.
D. Nothing; an empty report is always valid.

Answers.

B. use_structured_outputs_internally=True is the CodeAgent argument that makes each step’s output structured (it loads structured_code_agent.yaml and sets response_format per step). A is final-answer validation, not per-step parsing; C is dead (grammar= removed in 1.21.0); D only affects how many steps run.
B. As of smolagents 1.26.0, InferenceClientModel only honors response_format when the provider is in STRUCTURED_GENERATION_PROVIDERS = ["cerebras", "fireworks-ai"]; anything else raises a ValueError. The fixes are to pin one of those providers or switch to a backend with full support (OpenAIModel/LiteLLMModel).
C. A False (or a raise) is wrapped in an AgentError, caught, stored in ActionStep.error, and re-injected to the model via write_memory_to_messages(). The run loops and the model self-corrects — it does not crash. Only max_steps ends a run.
C. Bad arguments from the LLM raise AgentToolCallError, which descends from AgentExecutionError (not directly from AgentError). Like other in-step errors it’s captured in ActionStep.error and re-fed, so the model fixes the call next step. (AgentToolExecutionError is when the tool itself raises at runtime.)
C. Structured outputs guarantee form, never content — a typed QuillReport can be empty. Only check_has_chart (rejects chart_paths=[]) and check_has_source_for_web_claims (rejects empty sources after a web call) catch this. Form is not validity.

Common pitfalls.

Conflating the two “structured outputs.” response_format (a model setting, per call) is not use_structured_outputs_internally (a CodeAgent argument, per step) — and neither validates content. This is the number-one mistake in this area.
Copying the 2-arg check. The guided tour’s is_integer(final_answer, agent_memory=None) only works because the third argument is passed as a keyword. Write the 3-arg form (final_answer, memory, agent) so your checks are robust.
Writing grammar=. It was removed in smolagents 1.21.0. Plenty of tutorials still show it. The modern path is response_format=.

Key takeaways

smolagents has two structured-output mechanisms — response_format (model level, per call) and use_structured_outputs_internally (a CodeAgent argument, per step) — they are different, and neither validates the answer’s content.
final_answer_checks are 3-arg validators (final_answer, memory, agent); a False (or a raised exception) does not crash — it loops the agent so it self-corrects.
The AgentError hierarchy has seven classes; AgentToolCallError and AgentToolExecutionError descend from AgentExecutionError, and an in-step error lands in ActionStep.error and is re-fed to the LLM. Only max_steps ends a run (state="max_steps_error").
InferenceClientModel + response_format works only with provider in {"cerebras","fireworks-ai"} as of smolagents 1.26.0 — route through make_model() so you can swap backends.
grammar= is dead (removed in 1.21.0); use response_format=. agent.logs is gone (1.21.0); read agent.memory.steps.
Form is not validity. A perfectly-typed QuillReport can still have chart_paths=[] and sources=[] — only a final_answer_check enforces the business rule.

Want the full smolagents concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Next: Module 9 — Tool Interop: MCP, the Hub, and Other Ecosystems. Quill’s tools are still its own. In Module 9 you’ll plug it into the wider ecosystem with MCP and the Hub — and meet a different kind of structured output: an MCP tool’s output_schema, which (unlike final_answer_checks) is purely informational.

Links: Module 8 lab code · ← Module 7 · Module 9 → · Course index

References

smolagents docs — Agents reference (final_answer_checks, RunResult, MultiStepAgent). API is “experimental, subject to change.”
smolagents docs — Models reference (response_format, structured generation, custom models).
smolagents source v1.26.0 — src/smolagents/agents.py (_validate_final_answer, use_structured_outputs_internally) and src/smolagents/utils.py (the AgentError hierarchy, parse_code_blobs, truncate_content).
smolagents docs — How do multi-step agents work? (the ReAct loop and self-correction).
smolagents docs — Inspecting an agent run (the “LLM dumb errors” framing; reading agent.memory.steps).