aifromzero.dev

Vision and Multimodal: Quill Reads Charts and the Web (smolagents, Module 11)

Module 11 of 15 13 min read Lab code ↗

This is Module 11 of smolagents Mastery, a free 15-module course that takes you from your first code agent to a production-grade one. Start at Module 1 or browse the full syllabus.

The Quill you finished in Module 10 writes a chart and walks away. It loads your CSV, writes pandas, draws matplotlib, calls save_chart to drop a PNG in outputs/, and stitches the path into a QuillReport. Then it moves on — without ever looking at the pixels. It trusts its own code.

That trust bites on real analysis. The Y-axis starts at 80 and turns a 3% wiggle into a cliff-edge “explosion.” Two series overplot into an unreadable smear. A legend spills off the canvas. The report ships, the chart lies, and nobody catches it — least of all Quill, who has never seen a single pixel it produced. And some web sources give a text scraper nothing: a JavaScript dashboard, an interactive chart, a canvas plot. VisitWebpageTool comes back with markup and no data.

This module gives Quill eyes. It re-reads its own charts with a vision model — catching the truncated axis before the report goes out — and, optionally, it can see a page that a text scraper cannot read.

In this module

You’ll learn

  1. Feed images to an agent the right way — and tell the two channels apart: run(images=[...]) (vision content the model looks at) versus additional_args={...} (named Python variables the sandbox operates on).
  2. Choose a vision-capable model, and understand why flatten_messages_as_text=False is what actually keeps your image blocks alive on the way to the model.
  3. Build a screenshot step_callback that injects a PNG into observations_images each step — and prunes old screenshots so context (and cost) stays bounded.
  4. Debunk the naming trap: the smolagents[vision] extra installs a browser (helium + selenium), not image-input support — image input needs only a VLM.
  5. Recognize when vision earns its keep (chart self-review, JS-heavy pages) versus when a text tool is cheaper and just as good.

You’ll build Quill gains eyes: it re-reads its own charts with a VLM via run(images=[...]), and optionally browses JS-heavy pages with a vision_browser sub-agent.

Theory areas covered T11 — Multimodal & agentic RAG — 7% (this module owns the vision half; the RAG half is Module 12). (No “exam domain” — this course has no certification.)

Prerequisites Module 3 (Tool, save_chart, Agent Types), Module 4 (make_model, the Model classes), Module 6 (step_callbacks, ActionStep, memory), Module 10 (manager + web_researcher, managed_agents, additional_args). An HF_TOKEN in .env, and a VLM accessible (see the lab).

Where you are in the course

M1 ✅  M2 ✅  M3 ✅  M4 ✅  M5 ✅  M6 ✅  M7 ✅  M8 ✅  M9 ✅  M10 ✅
👉 M11  ⬜ M12  ⬜ M13  ⬜ M14  ⬜ M15

Quill so far: a manager CodeAgent that delegates web research to a web_researcher sub-agent, and writes charts blind. After this module: Quill re-reads its own charts with a VLM (and, optionally, gains a vision_browser sub-agent). The chain stays linear. M11 reuses two hooks you already built: the step_callback pattern from Module 6 (here in its full screenshot form) and the additional_args channel from Module 10.

Two ways to give an agent an image (and why the difference matters)

This is the heart of the module, so get it right before anything else. There are two ways to hand an image to a smolagents agent, and they are not interchangeable. Confuse them and your vision feature silently does nothing.

Here is the full signature of MultiStepAgent.run as of smolagents 1.26.0:

agent.run(
    task,
    stream=False,
    reset=True,
    images=None,            # ← Channel 1: vision content the model LOOKS AT
    additional_args=None,   # ← Channel 2: named Python variables in the sandbox
    max_steps=None,
    return_full_result=None,
)

Two of those parameters can carry an image. They do completely different things.

Channel 1 — run(images=[pil_image, ...]). run(images=...) injects the images as vision content into the message stream sent to the model. This is the “look at these images and answer” channel. The model sees the pixels. Under the hood, smolagents files them onto a TaskStep — its task_images: list[PIL.Image.Image] | None field — and writes them into the chat messages as image content blocks. This is the channel you want for “is this chart misleading?”

Channel 2 — additional_args={"chart": path_or_pil, ...}. additional_args are named Python variables placed into the CodeAgent’s sandbox. The model does not see the image. Instead, the variable exists in the executor namespace, and the model writes code that operates on it — chart.size, chart.resize(...), passing it to a tool. The default system prompt even shows the pattern ({'question': ..., 'image': 'path/to/image.jpg'}) and the docs add one instruction: “Give them clear names!” This is the channel for “load this image and compute on it.”

The distinction in one line: images= is vision content the model looks at; additional_args= is a named variable the sandbox operates on. For “is this chart misleading?”, you want the model to look — so images=. For “load this image and resize it”, you want a variable — so additional_args=. The two can coexist in the same call.

run(images=[...])additional_args={...}
What reaches the modelVision content (image blocks)Nothing visual — a named Python variable
Where it livesThe message stream (a TaskStep.task_images block)The sandbox / executor namespace
The model…sees the pixels (if it is a VLM)writes code that manipulates the object
Use it for”Look and answer” — critique a chart”Load and compute” — resize, hash, pass to a tool
Quill exampleRe-read a saved chart for honestyHand a CSV/image path down to a tool to process

If you read Module 10, additional_args is not new. It is the exact channel by which a manager CodeAgent passes rich objects down to a sub-agent — the auto-generated signature you saw was def web_researcher(task: str, additional_args: dict[str, Any]) -> str. The docstring there spells out the use: “Dictionary of extra inputs (e.g. images, dataframes).” M11 reuses that same hook. Same plumbing, applied to vision.

⚠️ Common misconception: “Passing an image in additional_args lets the model see it.”

It does not. additional_args puts a variable in the sandbox. If the model is not a VLM, or you never show it the image via images=, it sees nothing — it can only write code that touches the object. For the model to actually look at an image, you need images= and a VLM. This is the single most expensive mistake in smolagents vision, and it fails silently: no error, just an agent answering as if the image were invisible — because to it, it is.

Quill’s chart self-review is built on the right channel. Here is the core of review_charts from quill/agent.py — it loads each saved PNG and asks a VLM to critique it through images=:

def review_charts(report, *, model=None, prompt=CHART_REVIEW_PROMPT):
    if not report.chart_paths:
        return report  # nothing to review — no model call at all

    # A bare reviewer agent: no data tools, it only looks at the image and answers.
    reviewer = CodeAgent(
        tools=[],
        model=model or make_model(role="reviewer"),  # must resolve to a VLM
        managed_agents=[],
        max_steps=2,
    )

    for path in report.chart_paths:
        image = _load_chart_image(path)  # path -> a PIL image
        # THE channel: images= (vision content the model looks at), NOT additional_args.
        verdict = reviewer.run(prompt, images=[image], reset=True)
        report.caveats.append(f"Chart review ({path}): {verdict}")

    return report

Three details worth your attention. First, the image goes through images=[image] — never additional_args. Second, the verdict lands in report.caveats, an existing QuillReport field — no schema change (the M8 contract is frozen). Third, this is a separate, bare agent with tools=[]: re-reading a chart is an end-of-run sanity check, not part of the analysis loop. A VLM call is expensive, so you do it once at the end, not on every step.

Vision needs a VLM — and flatten_messages_as_text is the silent gatekeeper

You have the right channel. It still does nothing unless the model can process images. Vision is a property of the model, not of smolagents.

A VLM (vision-language model) is a model that accepts image content alongside text. The moment the underlying model is a VLM, run(images=[...]) works — no flag, no extra, no special agent. Verified VLM options as of smolagents 1.26.0:

  • InferenceClientModel(model_id="Qwen/Qwen2-VL-72B-Instruct") — the VLM the official vision-browser example uses.
  • gpt-4o / Claude 3.5 Sonnet via LiteLLMModel or OpenAIModel.

In Quill, you never instantiate these by hand. The model always comes through make_model (the frozen Module 4 contract). Point it at a VLM with one environment variable — no agent-code edit:

# Image input needs ONLY a VLM — no [vision] extra, no special install.
export QUILL_MODEL_ID="Qwen/Qwen2-VL-72B-Instruct"   # re-verify availability on Inference Providers
uv run python -m quill "Analyze data/sales.csv and chart monthly revenue, then check the chart yourself." --review

# Or swap the whole backend to OpenAI's gpt-4o — still one line, no code change:
# export QUILL_MODEL_BACKEND=litellm QUILL_MODEL_ID="gpt-4o"

The silent gatekeeper: flatten_messages_as_text

Here is where vision fails for people who did everything else right. Every Model class carries a flag, flatten_messages_as_text. When it is False, structured multimodal content — your image blocks — is preserved in the messages. When it is True, the messages are flattened to plain text, and the images are dropped on the way to the provider.

The good news: False is the default for InferenceClientModel, OpenAIModel, and AmazonBedrockModel. Vision just works on those.

The trap: LiteLLMModel and LiteLLMRouterModel set flatten_messages_as_text=True by default for model IDs that start with ollama, groq, or cerebras. So if you do vision via LiteLLM on one of those prefixes, your images can vanish — no exception, no warning, just a model answering as if the image were never there. If you need vision on those backends, set flatten_messages_as_text=False explicitly (and confirm the model itself is a VLM), or use a class that preserves image blocks by default.

Model classflatten_messages_as_text defaultImage blocks
InferenceClientModelFalsePreserved ✅
OpenAIModelFalsePreserved ✅
AmazonBedrockModelFalsePreserved ✅
LiteLLMModel (ollama/groq/cerebras prefix)TrueDropped ⚠️
LiteLLMModel (other models)FalsePreserved ✅

The headline trap: smolagents[vision] is a browser, not image input

This is the freshness flag of the module, and it trips a lot of tutorials. The extra smolagents[vision] installs helium + selenium — browser automation for the vision-web-browser example. It does not install image-input support.

Image input (run(images=...)) depends only on a VLM. It needs no extra at all. Pillow — which decodes a chart PNG into a PIL image — already ships with the data stack. Many readers pip install 'smolagents[vision]', expecting it to unlock images, and are baffled when their chart review still fails. It fails because they pointed a text model at it, not because they missed an install.

Quill’s lab makes the point concrete: the mandatory chart-review path installs no [vision] extra. You only add [vision] for the optional browser, later in this module.

# From quill/agent.py — Pillow is all the image INPUT needs (it ships with the data stack):
def _load_chart_image(path: str):
    import os
    from PIL import Image                # no [vision] extra required for image input
    if not os.path.exists(path):
        raise ValueError(f"No chart to review at {path!r}. ...")
    return Image.open(path).copy()       # .copy() detaches it from the file handle

When a tool returns an image: AgentImage

One more piece you met in Module 3, seen here through a vision lens. When a tool or an agent answer produces an image, smolagents wraps it in AgentImage — a type that behaves like a PIL.Image.Image and adds a few methods. The wrapping and unwrapping happen automatically through handle_agent_output_types / sanitize_inputs_outputs. The two methods that matter for vision:

from PIL import Image as PILImage
from smolagents import AgentImage

img = AgentImage(PILImage.new("RGB", (4, 4), "blue"))
raw = img.to_raw()         # -> the underlying PIL.Image.Image (ready for images=[...])
path = img.to_string()     # -> a path to the serialized image on disk
# img.save("out.png") also works

The practical payoff: if a tool hands back an image, you can re-feed it to a VLM cleanly — to_raw() gives you the PIL object that run(images=[img.to_raw()]) expects. No manual byte-wrangling.

Letting Quill see a page: the screenshot step_callback

Chart self-review is the mandatory half of vision. The optional half is the vision browser: a way to read pages that defeat a text scraper — JS dashboards, canvas charts, image-only content.

The canonical smolagents pattern is a CodeAgent that drives helium (over selenium, Chrome) inside its sandbox, plus a step_callback that screenshots the page after every step and injects it into memory so a VLM can read it on the next step. Quill’s optional vision_browser is exactly this.

You already built the step_callback mechanism in Module 6 — there you pruned big DataFrame dumps from observations. This module cashes that reuse in. A step_callback has the frozen smolagents signature (memory_step, agent) and runs inside the agent’s _finalize_step() (dispatched by a CallbackRegistry) after each step. Because it receives the agent, it can read and mutate agent.memory.steps — rewriting history before the next write_memory_to_messages() call.

step_callbacks accepts two shapes:

  • A list — step_callbacks=[fn] — fires on every step (the callbacks themselves short-circuit on the step types they ignore). This is what the vision browser uses.
  • A dict by type — step_callbacks={ActionStep: fn} — fires only for that step type.

The screenshot lands in a specific field. An ActionStep carries model_output, code_action, observations, observations_images, tool_calls, error, token_usage, and is_final_answer. The PNG goes into observations_images: list[PIL.Image.Image] | None — not the text observations string. The text observations carry the URL in words; the image channel carries the pixels a VLM reads on the next step.

Here is the screenshot callback from quill/callbacks.py, the full vision form of the M6 pattern:

from smolagents import ActionStep

def save_screenshot(memory_step: ActionStep, agent) -> None:
    if not isinstance(memory_step, ActionStep):
        return
    from time import sleep
    sleep(SCREENSHOT_SETTLE_SECONDS)        # 1.0s — let JS animations / lazy images settle

    png, current_url = _screenshot_png()    # helium's live Chrome, or (None, None)
    if png is not None:
        memory_step.observations_images = [_png_to_pil(png)]   # the VLM reads this NEXT step
        prune_old_screenshots(memory_step, agent)              # the cost guard

    if current_url is not None:
        existing = memory_step.observations or ""
        memory_step.observations = existing + f"\nCurrent url: {current_url}"

A note on imports: the official example imports from smolagents.agents import ActionStep. ActionStep is also exposed at the top level (from smolagents import ActionStep) as of smolagents 1.26.0 — Quill uses the top-level form. Both bind the same class.

The pruning loop — why it is not optional

The most important line in vision is the one that deletes screenshots. Here is prune_old_screenshots:

KEEP_LAST_SCREENSHOTS = 2

def prune_old_screenshots(memory_step: ActionStep, agent) -> None:
    if not isinstance(memory_step, ActionStep):
        return
    current = memory_step.step_number
    for step in agent.memory.steps:
        if not isinstance(step, ActionStep):
            continue
        # Prune any step now KEEP_LAST_SCREENSHOTS or more behind the current one.
        if step.step_number <= current - KEEP_LAST_SCREENSHOTS:
            step.observations_images = None

Why this matters: every step, the agent rebuilds its memory into messages with write_memory_to_messages() and re-sends them. Anything still sitting in an old observations_images is re-sent to the VLM on every later step. And a screenshot is the single most expensive thing in a VLM context — one image can cost as much as hundreds of text tokens. A 20-step browse that never prunes would send 20 images’ worth of tokens on every step. Pruning caps it: keep the last couple of screenshots (the agent usually needs the page it just acted on and the one before), null the rest. This is the same context-engineering idea as the M6 DataFrame pruning, applied to the observations_images channel.

The flow, end to end:

flowchart TD
    A[ActionStep N finishes] --> B["_finalize_step()"]
    B --> C["CallbackRegistry calls<br/>save_screenshot(memory_step, agent)"]
    C --> D["helium driver.get_screenshot_as_png()"]
    D --> E["memory_step.observations_images = [PNG as PIL]"]
    E --> F["prune_old_screenshots:<br/>null observations_images on steps ≤ N-2"]
    F --> G["Step N+1: VLM reads the image<br/>via write_memory_to_messages()"]
    C -.same callback hook as Module 6.-> B

Wiring the vision browser

The vision_browser is built in quill/team.py. It is a CodeAgent (it composes Python to drive a browser), tooled with three simple @tool helium helpers, with "helium" authorized in its sandbox and save_screenshot wired as a step callback:

def build_vision_browser(model=None, *, max_steps=VISION_BROWSER_MAX_STEPS):
    return CodeAgent(
        tools=vision_browser_tools(),            # go_back, close_popups, search_item_ctrl_f
        model=model or make_model(role="vision"),  # MUST be a VLM for a real browse
        name=VISION_BROWSER_NAME,                # exactly "vision_browser" (06 §2)
        description=VISION_BROWSER_DESCRIPTION,   # name + description => callable by the manager
        additional_authorized_imports=["helium"],  # its code does `from helium import *`
        step_callbacks=[save_screenshot],        # shoot + prune every step
        max_steps=max_steps,                     # 15 — every step is an LLM call AND a screenshot
    )

The tools are thin — go_back, close_popups, search_item_ctrl_f. The agent writes raw helium calls (go_to(...), click(...), scroll_down(...), Link(...), Text(...).exists()) directly in its sandboxed code, so the toolbox stays tiny. Before the first run, the reader preloads helium into the executor:

agent.python_executor("from helium import *", agent.state)

This is not done at construction on purpose — it would open a Chrome window the moment you build the agent.

One hard constraint, inherited from Module 5 and Module 10: the vision_browser drives a real local Chrome, so it stays executor_type="local". helium needs a local browser, and a remote executor plus managed_agents raises the documented exception you saw in M10. Running the whole team inside a sandbox (Approach 2) lifts that constraint — that is the capstone, Module 15.

The webagent CLI (and why its default is GPT-4o)

smolagents ships the vision-web-browser example packaged as a command: webagent (entry point smolagents.vision_web_browser:main). It is worth knowing because its defaults catch people out.

The flags and their values, as of smolagents 1.26.0:

FlagDefaultNotes
prompt (positional)a built-in search_requestoptional
--model-type"LiteLLMModel"
--model-id"gpt-4o"a VLM, via LiteLLM
--provider / --api-base / --api-key—provider routing

The browser opens Chrome, non-headless, in a 1000×1350 window, with the PDF viewer disabled.

Why gpt-4o? Vision navigation wants a strong, reliable VLM, and gpt-4o via LiteLLMModel is the packaged default. That choice carries a requirement: it needs the [litellm] extra and an OpenAI key in your environment. Run webagent without one and you get an OpenAI-key error.

Note the inconsistency, because it is itself a freshness flag: the example notebook uses Qwen/Qwen2-VL-72B-Instruct via InferenceClientModel, while the CLI defaults to gpt-4o via LiteLLM. Two different defaults across two surfaces. Do not read gpt-4o as “the vision model” — it is a swappable CLI default:

# Swap the CLI to a Hugging Face VLM instead of the gpt-4o default:
webagent "Find the latest figure on this page" \
  --model-type InferenceClientModel --model-id Qwen/Qwen2-VL-72B-Instruct

Distinguish it from the general-purpose smolagent CLI (a multi-step CodeAgent, covered in Module 13): smolagent is the generalist; webagent is the vision browser.

A cost note, honestly: gpt-4o is paid, and the HF free tier ($0.10/month, as of smolagents 1.26.0, subject to change) does not stretch far across a 72B VLM either. Vision is not where you run unbounded experiments.

Build it: Quill re-reads its own charts

The lab lives in full at smolagents-course-labs/module-11. It builds on the cumulative Module 10 state, and its tests pass offline — no token, no network, no VLM, no Chrome.

The goal: Quill writes its chart as before, then re-reads it with a VLM as a sanity check, and the verdict lands in QuillReport.caveats. The headline command (image input needs only a VLM — note the absence of any [vision] install):

QUILL_MODEL_ID="Qwen/Qwen2-VL-72B-Instruct" \
  uv run python -m quill "Analyze data/sales.csv and chart monthly revenue, then check the chart yourself." --review

Expected output:

[Quill] Backend: hf | Model: Qwen/Qwen2-VL-72B-Instruct
...                        # ← Quill writes pandas, draws matplotlib, save_chart -> outputs/monthly_revenue.png
===== CHART SELF-REVIEW (VLM re-reads the charts via run(images=...)) =====
  Chart review (outputs/monthly_revenue.png): Y-axis starts at 80, which exaggerates the trend — recommend starting at 0.
===== REPORT =====
# Analyze data/sales.csv and chart monthly revenue

## Findings
- Monthly revenue rises steadily through 2025.

## Charts
- `outputs/monthly_revenue.png`

## Caveats
- Chart review (outputs/monthly_revenue.png): Y-axis starts at 80, which exaggerates the trend — recommend starting at 0.

The wiring in build_quill is a pure addition — one new keyword-only argument, browse: bool = False, so every prior call site is unchanged. The default team gains the vision_browser only when you ask for it:

# From quill/agent.py — browse=True ADDS the vision_browser to the default team (stays local).
if managed_agents is _DEFAULT_TEAM:
    resolved_managed_agents = [build_web_researcher(model=model)]
    if browse:
        resolved_managed_agents.append(build_vision_browser(model=model))
        # # M15: Approach 2 (whole team in a sandbox) lifts the local-only constraint.

The critique prompt steers the VLM at the real chart lies, so the verdict is actionable:

CHART_REVIEW_PROMPT = (
    "You are reviewing a data chart you just produced, as a careful data analyst. Look at the "
    "image and judge whether it is HONEST and READABLE. Flag specifically: a Y-axis that does "
    "not start at zero and so exaggerates a trend; overlapping/overplotted series; an unreadable "
    "or off-canvas legend or labels; a misleading dual axis; or a wrong chart type for the data. "
    "Answer in ONE or TWO sentences ... Be concise — your answer becomes a caveat on the report."
)

The CLI also exposes --review (chart self-review, on by default for python -m quill.agent) and --browse (the optional vision browser):

# Optional vision browser — needs a local Chrome + pip install 'smolagents[vision]' + a VLM:
QUILL_MODEL_ID="Qwen/Qwen2-VL-72B-Instruct" uv run python -m quill "..." --browse

Run the offline suite to confirm the wiring without spending a token:

uv run pytest module-11/tests/                    # offline — no token, no network, no VLM, no Chrome
QUILL_LIVE_TESTS=1 uv run pytest module-11/tests/ # + the real VLM chart re-read (needs HF_TOKEN + a VLM)

The offline tests prove the channel: a spy fake model records the messages it is given, and the test asserts the chart PNG arrives as an image content block via run(images=...), never as an additional_args sandbox variable. The screenshot tests inject a fake PIL image through a patched driver seam — no real browser — and assert that save_screenshot injects into observations_images and prunes steps ≤ N-2.

Try it yourself. (1) Give Quill two versions of the same chart — one with a truncated Y-axis (plt.ylim(80, ...)) and one starting at zero — and ask which one misleads. Check that the VLM flags the truncation. (2) Run webagent "..." --model-type InferenceClientModel --model-id Qwen/Qwen2-VL-72B-Instruct on a chart-heavy page and compare quality, cost, and latency to the gpt-4o default.

In production

Vision costs real money. A single image is often worth hundreds of tokens, so a run that re-reads four charts and a browser that screenshots ten steps burns the free tier ($0.10/month, as of smolagents 1.26.0) fast. That is why the screenshot callback prunes old shots — without it, the per-step cost grows every step — and why the browser carries a max_steps cap (15 in Quill). Latency follows: a 72B VLM or gpt-4o is slower than a text model, so you re-read at the end of a run as a sanity check, not on every step.

Reliability is the other half. A helium/selenium browser is fragile — popups, CAPTCHAs, asynchronous JS (which is why the callback sleep(1.0)s before each shot). In production I cap browser retries at 2; beyond that you are paying VLM tokens to hammer a broken page. Keep a graceful degradation path: fall back to the text web_researcher from Module 10 when the vision browse fails.

Security carries over from Module 5: the vision_browser executes code (helium) and visits arbitrary URLs, so prompt injection via page content is a live risk. The full hardening — the entire system inside a remote sandbox (Approach 2) — is the capstone (Module 15). And choose vision when it pays: a chart to validate, a JS page a scraper can’t read. For plain text, VisitWebpageTool is cheaper and just as good.

Concept check

Why this matters. Vision in smolagents hinges on three things people routinely get wrong: which channel shows the model an image (images=, not additional_args=), what actually enables vision (a VLM, not the [vision] extra), and how to keep screenshots from blowing up your token bill (prune observations_images). These five scenarios drill exactly those distinctions.

1. A developer wants the agent to answer a question about the content of a chart — “does this chart exaggerate the trend?” Which call gets the model to actually look at the chart?

  • A. run(task, additional_args={"chart": pil_image})
  • B. run(task, images=[pil_image])
  • C. Install smolagents[vision], then pass the path in the task string
  • D. Set flatten_messages_as_text=True and pass the image in additional_args

2. “My chart review fails to see the image, even though I ran pip install 'smolagents[vision]'.” What is the most likely cause?

  • A. [vision] installs helium + selenium (a browser), not image input — the model is probably not a VLM
  • B. [vision] was installed but not imported
  • C. Image input requires smolagents[image], a different extra
  • D. The image must be base64-encoded before run(images=...)

3. A developer uses a VLM through LiteLLMModel with a model id starting ollama/..., and the images never reach the model. What is happening?

  • A. LiteLLMModel cannot do vision at all
  • B. flatten_messages_as_text defaults to True for ollama/groq/cerebras, flattening the image blocks to text
  • C. Ollama strips images server-side and there is no fix
  • D. images= is ignored unless stream=True

4. On a vision browser, why does the save_screenshot callback null out observations_images on older steps?

  • A. To free disk space where PNGs are written
  • B. Because old screenshots are re-sent to the VLM every step, and each image costs hundreds of tokens — pruning bounds the cost
  • C. Because ActionStep can hold only two images total
  • D. To force the agent to re-navigate to earlier pages

5. A reader runs webagent "summarize this dashboard" and gets an OpenAI-key error. Why, and what is the fix?

  • A. webagent is broken in 1.26.0; use smolagent instead
  • B. webagent defaults to gpt-4o via LiteLLMModel (needs [litellm] + an OpenAI key); swap with --model-type InferenceClientModel --model-id Qwen/Qwen2-VL-72B-Instruct
  • C. webagent always requires an OpenAI key; there is no alternative
  • D. The HF free tier expired; pay for Inference Providers

Answers.

1 — B. run(images=[...]) injects the image as vision content the model looks at (it becomes a TaskStep.task_images block in the message stream). additional_args (A, D) only puts a variable in the sandbox — the model never sees the pixels. The [vision] extra (C) is a browser, irrelevant to image input.

2 — A. smolagents[vision] installs helium + selenium (browser automation), not image-input support. Image input needs only a VLM and Pillow (which ships with the data stack). The symptom — installing [vision] and still seeing nothing — almost always means the model isn’t a VLM.

3 — B. LiteLLMModel/LiteLLMRouterModel set flatten_messages_as_text=True by default for model ids starting ollama, groq, or cerebras. That flattens structured content (your image blocks) to plain text, silently dropping the images. Set it to False explicitly (and confirm the model is a VLM), or use a class that preserves image blocks by default.

4 — B. Memory is rebuilt into messages with write_memory_to_messages() every step, so anything in an old observations_images is re-sent to the VLM each step. One image can cost hundreds of text tokens; without pruning, a 20-step browse sends 20 images’ worth of tokens per step. Keeping only the last couple bounds it.

5 — B. As of smolagents 1.26.0, webagent defaults to --model-type LiteLLMModel --model-id gpt-4o, which requires the [litellm] extra and an OpenAI key. It is a swappable default — point it at a Hugging Face VLM with --model-type InferenceClientModel --model-id Qwen/Qwen2-VL-72B-Instruct.

Common pitfalls

  • Installing smolagents[vision] to “enable images.” It installs a browser (helium + selenium). Image input depends only on a VLM — no extra at all. This is the number-one stale-tutorial mistake in smolagents vision.
  • Putting an image in additional_args and expecting the model to see it. That is a sandbox variable; the model can only write code that touches it. For the model to look, use images= — and a VLM.
  • Forgetting flatten_messages_as_text and forgetting to prune. Vision via LiteLLMModel on ollama/groq/cerebras flattens messages (True by default) and drops images silently. And screenshots you never prune re-send on every step — cost and context grow without bound.

Key takeaways

  • images= is vision content the model looks at; additional_args= is a named variable the sandbox operates on. For “is this chart misleading?”, use images=.
  • Vision is a property of the model (a VLM) — no smolagents extra is required for image input.
  • The smolagents[vision] extra installs helium + selenium (a browser), not image input. They are two different things.
  • flatten_messages_as_text=False preserves image blocks (the default for InferenceClientModel/OpenAIModel/AmazonBedrockModel); True drops them — and it is the default for LiteLLMModel on ollama/groq/cerebras.
  • A screenshot is injected into observations_images via a step_callback with the signature (memory_step, agent) — and you must prune old screenshots (each image ≈ hundreds of tokens).
  • A tool-produced image is wrapped in AgentImage; to_raw() gives back the PIL image for images=[...], to_string() gives a path.
  • Vision is expensive — use it when it pays (chart self-review, JS-heavy pages), not by default.

What’s next

Quill can now read its own charts and, optionally, the web — but it still guesses what your columns mean. Module 12 grounds it in a knowledge base: a RetrieverTool so Quill looks up and cites the meaning of your data instead of inventing it.

Want the full smolagents concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Module 11 lab code · ← Module 10 · Module 12 → · Course index

References