Durable Execution, Resilience, and Sandboxing (LangGraph in Production, Module 09)

This is Module 09 of LangGraph in Production, a free 16-module course that takes you from your first StateGraph to a deployed, durable autonomous coding agent on LangGraph v1.x. Start at Module 1 or browse the full syllabus.

---

The problem with “good enough”

Forge works. You built it across eight modules: it plans, edits files in parallel, runs tests in a fix-it loop, checkpoints state to SQLite so it survives a kill, lets you approve the plan and the PR, and even lets you rewind time with get_state_history. That is a genuine production agent.

But “survives a kill” is not the same as “resilient.” Consider this scenario.

Forge is fixing the Tally issue: “Bug: tally report --month crashes when no expenses for a category.” The Planner calls list_files then read_file on four Tally source files — five model calls total. The patch is correct on the first try. The Tester runs pytest in its subprocess sandbox. That call fails: OSError: [Errno 26] Text file busy. That is a transient race condition your CI runner gets once a month. Forge does not know that. It sees TestResult(passed=False), hands control to the Fixer, runs the Fixer and the Tester again, and again. Three extra model calls, no actual bug to fix. Cost of flakiness: nine model calls instead of three.

That is not the worst part. The next issue on your queue contains a FileEdit whose new_content is a perfectly valid-looking Python file — except it opens ../../.env to read your API keys. The apply_edits node writes that content right where the LLM said to put it, no questions asked. Forge executes model-generated code. An LLM can hallucinate a path traversal. Nothing in Forge stops it.

This module fixes both: Forge becomes resilient (the Tester retries transient failures in place) and hardened (the sandbox blocks path traversal, limits subprocess runtimes, and strips secrets from the subprocess environment).

---

In this module

You’ll learn:

• Choose the right durability mode ("exit" / "async" / "sync") for a given Forge run and understand what each one guarantees after a crash.
• Attach a RetryPolicy to the Tester node so transient subprocess failures are absorbed automatically, without triggering the Fixer loop.
• Cache the repo analysis with a CachePolicy and InMemoryCache so the Planner does not re-read Tally on every run of the same issue.
• Understand idempotence with @task as the Functional API’s replay unit (a conceptual bridge to Module 13).
• Harden forge/sandbox.py against path traversal, subprocess hangs, and secret leakage.

You’ll build: Forge gains a RetryPolicy on the Tester, a CachePolicy on the Planner, the correct durability= mode for production use, and a forge/sandbox.py that blocks path traversal, times out runaway processes, and filters secrets from the subprocess environment.

Concepts covered: CA9 — Durable execution & resilience (core). This module BUILDs CA9.1–CA9.6 of the LangGraph theory inventory.

Prerequisites: Modules 1–8. The lab requires the cumulative forge/ package with the SqliteSaver checkpointer from Module 6 — durability builds on persistence. A .env with ANTHROPIC_API_KEY, or the free Ollama fallback (zero cost).

---

Where you are

M1 ✅  StateGraph, Forge + Tally
M2 ✅  ForgeState, build_graph()
M3 ✅  triage router, test-fix loop, sandbox
M4 ✅  Planner, create_agent, read tools
M5 ✅  fan-out via Send, Editor×N, fan-in
M6 ✅  SqliteSaver, threads, get_state, resume after kill
M7 ✅  interrupt() + Command(resume=), plan & PR approval
M8 ✅  get_state_history, replay, fork via update_state
M9 👉  durability modes, RetryPolicy, CachePolicy, sandbox hardening
M10 ⬜  the Store, long-term memory
M11–M16 ⬜

Forge before this module: survives a kill (M6), but not transient Tester failures; the Planner re-reads Tally on every run; no control over the durability checkpoint granularity; sandbox has no timeout, no path guard, no secret filter.

Forge after this module: Tester retries on transient errors without triggering the Fixer; Planner output is cached for 300 seconds per issue; durability mode is an explicit choice at invoke time; sandbox blocks path traversal, times out after 120 seconds, and never leaks API keys to the test subprocess.

What does NOT change in M9: zero new fields on ForgeState. The graph shape is identical to M7/M8 — M9 is purely additive on graph.py and sandbox.py.

In Module 10, Forge gains cross-thread long-term memory — the Store — so the Planner can recall how similar issues were resolved in the past.

---

Durability modes: choosing how durable your checkpoints are

You learned in Module 6 that the checkpointer writes a checkpoint after every superstep, which is what lets graph.invoke(None, config) on the same thread_id resume a killed run. But Module 6 used the default behavior. That default is one of three durability modes in LangGraph v1.x, and choosing the right one matters in production.

The three modes

The durability= parameter on invoke / stream controls when the checkpoint is flushed relative to the next node’s execution.

"exit" — checkpoint only at the end of the run (or at an interrupt()). Maximum throughput, minimum write amplification. If the process dies mid-run, the next invoke restarts from the beginning (or from the last interrupt() checkpoint, because HITL interrupts always checkpoint regardless of the durability mode). Use this for batch jobs with no human-in-the-loop and cheap-to-rerun work.

"async" — checkpoint is written during the execution of the next node (writes overlap with computation). Good balance of throughput and recoverability. This is the default and is what Forge has been using implicitly since Module 6.

"sync" — checkpoint is written before the next node starts. Maximum recoverability: if the process dies between any two nodes, the next invoke resumes from the last completed node. Maximum I/O overhead. Use this when nodes have irreversible side effects (writing files, opening a PR) and the cost of replaying those effects exceeds the I/O cost.

	"exit"	"async" (default)	"sync"
When checkpoint is flushed	End of run (or interrupt())	Concurrently with next node	Before next node starts
I/O cost	Lowest	Medium	Highest
Recovery granularity after crash	Beginning of run (or last interrupt())	Usually last completed node	Last completed node, guaranteed
Forge use case	Batch analysis with no side effects	Interactive agent sessions	Runs that edit files and open PRs

The v1.x syntax

# Preferred for a run that edits files and opens a PR
graph.invoke(input, config, durability="sync")

# Default for most interactive runs
graph.stream(input, config, stream_mode="updates", durability="async")

# Batch mode — fastest, lowest recoverability
graph.invoke(input, config, durability="exit")

Never write checkpoint_during=True or checkpoint_during=False in new code. That was the LangGraph v0.x API. In v1.x it is deprecated. The mapping is checkpoint_during=True → durability="async" and checkpoint_during=False → durability="exit", but the semantics are not identical across patch versions. Use durability= in all new code.

Resume after crash

The mechanics are unchanged from Module 6. After a crash, you call graph.invoke(None, config) on the same thread_id. The checkpointer replays state from the last saved checkpoint. What changes with the durability mode is the granularity of that checkpoint: with "exit", you may restart from the beginning of the run; with "sync", you restart from the node boundary immediately before the crash. In production, “I wasted 2 Planner calls” is not catastrophic, but “I opened the same PR twice” is — which is why "sync" is the right choice for Forge runs that modify files and open pull requests.

Here is the chronogram showing how the three modes differ:

gantt
    title Durability modes — checkpoint timing across two nodes
    dateFormat  ss
    axisFormat %S

    section "exit"
    Node A executes      :a1, 00, 3s
    Node B executes      :a2, 03, 3s
    Checkpoint flushed   :milestone, 06, 0s

    section "async" (default)
    Node A executes      :b1, 00, 3s
    Checkpoint flushed   :b2, 03, 1s
    Node B executes      :b3, 03, 3s

    section "sync"
    Node A executes      :c1, 00, 3s
    Checkpoint flushed   :c2, 03, 1s
    Node B starts        :c3, 04, 3s

With "sync", Node B only starts after the checkpoint for Node A has been durably written. With "async", writing overlaps with Node B’s execution. With "exit", nothing is written until the run is over.

The lab demonstrates all three modes with a real Forge run. The build_graph() function accepts the durability setting at invoke time — you do not bake it into the compiled graph.

---

RetryPolicy: making the Tester resilient to flakiness

The RetryPolicy is a LangGraph primitive that makes a node automatically retry on failure — without the graph ever knowing the failure happened. It absorbs transient errors within a single superstep, invisibly.

Import it from the correct v1.x location:

from langgraph.types import RetryPolicy

Attaching a RetryPolicy to the Tester

In forge/graph.py, the Tester (run_tests node) gets the policy at add_node time:

builder.add_node(
    "tester",
    nodes.tester,
    retry_policy=RetryPolicy(max_attempts=3, retry_on=Exception),
)

This is the exact code from the lab. Let’s unpack the parameters:

• max_attempts=3 — the node will be tried up to 3 times total: 1 original attempt + 2 retries. Beyond 3, the error propagates to the graph.
• retry_on=Exception — retry on any exception. You can narrow this to a specific exception type (e.g., OSError) or pass a callable (exc) -> bool for fine-grained control. The default when retry_on=None is to retry on everything except ValueError and TypeError, which are programming errors rather than transient failures.
• initial_interval (default 0.5) — seconds to wait before the first retry.
• backoff_factor (default 2.0) — exponential back-off multiplier. With initial_interval=0.5 and backoff_factor=2.0, the delays are 0.5s, 1.0s, 2.0s.
• jitter (default True) — adds random noise to the delay to prevent thundering-herd effects when many nodes retry simultaneously.

What RetryPolicy does NOT do

RetryPolicy does not update state. Between retries, the graph state is unchanged. The attempts counter in ForgeState — introduced in Module 3 as the guard for the test-fix loop — does not increment during a retry. This is fundamental: a retry is invisible to the graph. It is a local recovery mechanism.

RetryPolicy is not the same as the Tester → Fixer loop. The loop exists at the superstep level: it runs the Tester, observes the state, decides to route to the Fixer, which modifies files, then runs the Tester again. Each cycle is a new superstep with a new checkpoint. The RetryPolicy operates entirely within a single superstep, before any state update is written.

The test in tests/test_pipeline.py shows this concretely:

def test_retry_policy_recovers_transient_tester_failure(monkeypatch, make_routed_model):
    # Monkeypatch sandbox.run_tests to fail on the first call only
    real = sandbox.run_tests
    calls = {"n": 0}

    def flaky(repo_path, timeout=120):
        calls["n"] += 1
        if calls["n"] == 1:
            raise RuntimeError("transient subprocess failure")
        return real(repo_path, timeout)

    monkeypatch.setattr(sandbox, "run_tests", flaky)
    graph = build_graph(checkpointer=checkpoint.in_memory())
    cfg = {"configurable": {"thread_id": "retry"}, "recursion_limit": 50}

    graph.invoke({"issue": "...", "repo_path": repo}, cfg)      # -> plan_approval
    r2 = graph.invoke(Command(resume={"approved": True}), cfg)  # approve plan -> tester retries
    assert r2["__interrupt__"][0].value["action"] == "approve_pr"  # reached PR step
    assert calls["n"] == 2   # 1 failure + 1 success — no Fixer, no extra model calls

The Tester failed once, retried once, succeeded — and the graph went straight to PR approval. The Fixer was never invoked. Zero extra model calls. That is the whole point.

Why NOT to put RetryPolicy on the Fixer

The Fixer calls an LLM. LLMs are non-deterministic: the same input does not produce the same output. Retrying the Fixer on a transient error would produce a different FileEdit on each attempt and apply them all, leading to inconsistent state. A node with side effects that differ across calls should not have a RetryPolicy. The Fixer belongs to the Tester → Fixer cycle, not to a retry policy.

The test_module_09.py test suite captures the core mechanic directly:

def test_retry_policy_retries_until_success():
    calls = {"n": 0}

    def flaky(s):
        calls["n"] += 1
        if calls["n"] < 3:
            raise RuntimeError("transient boom")
        return {"value": "ok"}

    b = StateGraph(_S)
    b.add_node("n", flaky, retry_policy=RetryPolicy(max_attempts=3, retry_on=Exception))
    b.add_edge(START, "n")
    b.add_edge("n", END)
    out = b.compile().invoke({"value": ""})
    assert out["value"] == "ok" and calls["n"] == 3

The node fails twice and succeeds on the third attempt. The graph sees only the final success.

---

CachePolicy and InMemoryCache: stop paying to re-read the same repo

The Planner reads the Tally repository on every run. It calls list_files to enumerate source files, then read_file on each. For a four-file codebase, that is five tool calls before any actual editing begins. When you fix three issues in a row against the same Tally commit, you make those five calls three times. The repo has not changed. You are paying to re-read the same files.

This is a perfect use case for CachePolicy — LangGraph’s node-level caching primitive.

The imports

from langgraph.types import CachePolicy
from langgraph.cache.memory import InMemoryCache

Both must come from these exact paths. InMemoryCache is not InMemorySaver (which is the in-memory checkpointer from Module 6). They are different primitives: InMemorySaver persists graph state between runs; InMemoryCache memoizes node outputs within and across runs.

Attaching a CachePolicy to the Planner

builder.add_node(
    "planner",
    nodes.planner,
    cache_policy=CachePolicy(key_func=lambda s: s.get("issue", ""), ttl=300),
)

This is the exact code from forge/graph.py. The key_func takes the node’s input state and returns a string cache key. Here it uses the issue text — if Forge processes the same issue twice within 300 seconds (5 minutes), the second run will return the cached Plan without calling the Planner node at all.

The ttl parameter is in seconds. After the TTL expires, the next invocation runs the node again and caches the fresh result.

The compile() requirement

graph = builder.compile(checkpointer=checkpointer, cache=InMemoryCache())

compile(cache=...) is not optional. If you add cache_policy= to a node but forget cache=InMemoryCache() at compile time, the policy is silently ignored. No error, no warning. The node runs every time. This is a common trap.

In forge/graph.py, the build_graph() function passes the cache at compile time:

def build_graph(checkpointer=None):
    builder = StateGraph(ForgeState)
    # ... nodes and edges ...
    return builder.compile(checkpointer=checkpointer, cache=InMemoryCache())

The InMemoryCache is appropriate for development and labs. For production, LangGraph also provides a SqliteCache backend. You can swap backends without changing the node definitions — the cache_policy= declarations on the nodes stay the same.

The unit test

def test_cache_policy_skips_recomputation():
    calls = {"n": 0}

    def expensive(s):
        calls["n"] += 1
        return {"value": "computed"}

    b = StateGraph(_S)
    b.add_node("n", expensive, cache_policy=CachePolicy(key_func=lambda s: s["value"]))
    b.add_edge(START, "n")
    b.add_edge("n", END)
    graph = b.compile(cache=InMemoryCache())
    graph.invoke({"value": "k"})
    graph.invoke({"value": "k"})   # same cache key
    assert calls["n"] == 1         # second call was served from cache

Two invocations with the same cache key — only one execution. This is the property you rely on for the Planner.

Cache validity and idempotence

A node is a valid cache target if and only if it is idempotent: same input produces the same output, with no side effects that matter. The Planner reads the repo and produces a Plan — it does not write anything. For the same issue text and the same repo contents, it should produce the same Plan. That is a reasonable approximation within a 5-minute window.

The Fixer is not idempotent — every LLM call is different. The apply_edits node is not idempotent — it writes files. Do not cache those.

Three distinct storage primitives

Primitive	Module	Purpose	Scope
InMemorySaver / SqliteSaver	M6	Checkpointer — persists graph state per thread per superstep	Cross-run, thread-scoped
InMemoryCache	M9	Node cache — memoizes node outputs by cache key + TTL	Cross-run, graph-wide
the Store	M10	Long-term memory — semantic search, cross-thread recall	Cross-run, namespace-scoped

They are not interchangeable. The checkpointer stores the entire ForgeState; the cache stores individual node outputs; the Store stores arbitrary structured data you put there explicitly. Module 10 builds the Store.

---

Idempotence with @task: the Functional API’s replay unit

This section is a conceptual bridge to Module 13. You will not add any @task to Forge here — the build belongs to Module 13. But the concept explains why idempotence is fundamental to everything you have done in Module 9.

In Module 7, you learned that when an interrupt() node resumes, it re-executes from its beginning. That is why plan_approval must be side-effect free before the interrupt() call. The entire node reruns; only the interrupt() value changes.

The Functional API (@entrypoint / @task) has a finer-grained mechanism. A @task is the unit of replay: when an @entrypoint resumes after a crash or an interrupt(), the results of already-completed @task invocations are replayed from the checkpoint, not re-executed. This is more precise than node-level replay.

# Module 13 — shown here as a conceptual preview, not built in Forge yet
from langgraph.func import task
from langgraph.types import RetryPolicy

@task(retry_policy=RetryPolicy(max_attempts=3))
def run_tests_task(repo_path: str, edits: list) -> TestResult:
    ...  # same logic as the tester node, but as a @task

Notice that @task accepts the same retry_policy= and cache_policy= parameters as add_node. The primitives you learned in this module apply to both APIs.

Why does this matter now? Because RetryPolicy and CachePolicy both require idempotent targets. The @task decorator makes the idempotence requirement explicit: when LangGraph replays a @task, it expects to get the same result. If your task has irreversible side effects, replay will duplicate them.

The Tester (run_tests) is idempotent: subprocess.run(pytest) on the same repo contents produces the same TestResult. Caching it or retrying it is safe. The Fixer is not: model.invoke() varies. RetryPolicy on the Fixer would produce three different FileEdits, not one reliable one.

The Functional API is an alternative to the Graph API — not a replacement. Module 13 will compare them side by side and build the Forge variant. For now, hold onto the concept: @task is the unit of durable, idempotent replay in the Functional API.

---

Sandboxing and agentic security: Forge executes code

This is the section that most tutorials skip. It is the most production-relevant part of the module.

Forge is not just a chatbot. It calls write_file to apply FileEdit.new_content directly to disk, then launches pytest in a subprocess. The content of those files comes from an LLM. That means Forge is executing untrusted code on your machine.

That is not a theoretical concern. LLMs hallucinate. In 2026, several publicly disclosed vulnerabilities in agentic LLM systems involved code execution vectors that the agent authors had not anticipated — path traversal in file writes, subprocess calls that exfiltrated API keys through environment variable inheritance, tests that made outbound network connections. These are real attack surfaces when an agent writes and executes code.

Decision L8 in the course architecture mandates a hardened sandbox for exactly this reason.

The threat model for Forge

Path traversal (the write-outside-sandbox attack). An LLM generating a FileEdit might produce path = "../../.env". Without validation, sandbox.write_file(repo_path, "../../.env", "hacked") resolves outside the tempdir and overwrites your .env. This is a real-world path traversal attack, not a hypothetical.

Subprocess hang (the infinite loop attack). The LLM might generate test code — or the model-written source file might accidentally contain — an infinite loop. Without a timeout, subprocess.run(pytest) blocks indefinitely. The Tester node never returns. The graph hangs.

Secret exfiltration (the environment inheritance attack). By default, subprocess.run inherits the parent process’s entire environment, including ANTHROPIC_API_KEY, LANGSMITH_API_KEY, and any other secrets you have in your shell. Test code could read os.environ["ANTHROPIC_API_KEY"] and log it or post it to a remote endpoint. The subprocess should run with a filtered environment.

SSRF via test code. A model-generated test could import requests and make outbound calls to arbitrary URLs. In a production container you would disable network egress. In the OSS version, you document the gap and keep the subprocess environment minimal.

The hardened sandbox.py

The lab’s sandbox.py addresses three of the four vectors directly:

def run_tests(repo_path: str | Path, timeout: int = 120) -> TestResult:
    """Run the target package's pytest suite in an isolated subprocess with a timeout."""
    repo = Path(repo_path)
    root, pkg = repo.parent, repo.name
    cmd = [sys.executable, "-m", "pytest", pkg, "-q", "--no-header", "-p", "no:cacheprovider"]
    try:
        proc = subprocess.run(cmd, cwd=root, capture_output=True, text=True, timeout=timeout)
    except subprocess.TimeoutExpired:
        return TestResult(
            passed=False, total=0, failures=["<timeout>"],
            output_tail=f"pytest timed out after {timeout}s (possible infinite loop in the edit)",
        )
    out = proc.stdout + proc.stderr
    ...

The timeout is enforced by subprocess.run(..., timeout=timeout). When pytest exceeds 120 seconds, Python raises subprocess.TimeoutExpired. The catch converts this to a clean TestResult(passed=False) with a diagnostic message — the graph handles it like any other failing test result, and the RetryPolicy will not help (the timeout is not transient). The test in test_module_09.py verifies this:

def test_sandbox_times_out_on_runaway_test():
    repo = sandbox.make_sandbox(Path(__file__).resolve().parents[1] / "tally")
    sandbox.write_file(repo, "tests/test_slow.py",
                       "import time\n\ndef test_slow():\n    time.sleep(5)\n")
    result = sandbox.run_tests(repo, timeout=1)
    assert result.passed is False and "timed out" in result.output_tail

For path traversal, the write_file function should validate the target before writing. The production-hardened version uses pathlib.Path.resolve() to canonicalize the path (which resolves .. components and symlinks) and then checks that the resolved path is inside the sandbox:

def write_file_safe(repo_path: str | Path, rel: str, content: str) -> str:
    """Write a file inside the sandbox — blocks path traversal."""
    sandbox = Path(repo_path).resolve()
    target = (sandbox / rel).resolve()
    if not str(target).startswith(str(sandbox)):
        raise ValueError(f"Path traversal blocked: {rel!r} resolves outside sandbox")
    target.parent.mkdir(parents=True, exist_ok=True)
    target.write_text(content)
    return str(target)

Why resolve() before the startswith check? Because a simple string check on the unresolved path is bypassable. ../../etc/passwd would not start with the sandbox path as a string, but what about ./foo/../../../etc/passwd? And what about symlinks that point outside the sandbox? Path.resolve() follows symlinks and resolves all .. components to a canonical absolute path. Only then is the prefix check reliable.

For secret filtering, the subprocess environment should be cleaned before passing to pytest. The pattern is:

import os

def _filtered_env() -> dict:
    """Remove API keys and cloud credentials from the subprocess environment."""
    skip = ("ANTHROPIC_", "LANGSMITH_", "OPENAI_", "AWS_", "GOOGLE_")
    return {k: v for k, v in os.environ.items() if not k.startswith(skip)}

# Then in run_tests:
proc = subprocess.run(cmd, cwd=root, capture_output=True, text=True,
                      timeout=timeout, env=_filtered_env())

In Docker or LangGraph Platform (Module 14), you can go further: run the subprocess with --network none or use container-level network isolation so test code cannot reach the internet at all. For the OSS version running on your laptop, the filtered environment and the timeout are the primary defenses.

The sandbox architecture

graph TB
    subgraph "Forge process"
        AG[apply_edits node] --> WF[write_file_safe\npath traversal check]
        TN[tester node] --> RT[run_tests\ntimeout=120s]
    end

    subgraph "Sandbox temp dir"
        WF --> SC[sandbox copy of Tally\n/tmp/forge-xxxxx/tally/]
        RT --> PT[pytest subprocess\nfiltered env, no secrets]
    end

    SC --> PT
    PT --> TR[TestResult\npassed / failures / output_tail]
    TR --> TN

    style SC fill:#e8f5e9
    style PT fill:#fff3e0

The original Tally repo is never touched. make_sandbox copies it to a fresh tempfile.mkdtemp() on every Forge run. All writes go to that copy. The pytest subprocess runs inside that copy with a filtered environment and a hard timeout.

---

Putting it all together: the M9 graph

Here is the complete build_graph() from the lab — every resilience primitive in place:

from langgraph.cache.memory import InMemoryCache
from langgraph.graph import END, START, StateGraph
from langgraph.types import CachePolicy, RetryPolicy

from forge import nodes
from forge.state import ForgeState


def build_graph(checkpointer=None):
    builder = StateGraph(ForgeState)
    builder.add_node("intake", nodes.intake)
    builder.add_node("triage", nodes.triage)
    builder.add_node(
        "planner", nodes.planner,
        cache_policy=CachePolicy(key_func=lambda s: s.get("issue", ""), ttl=300),
    )
    builder.add_node("plan_approval", nodes.plan_approval)
    builder.add_node("abandon", nodes.abandon)
    builder.add_node("editor", nodes.editor)
    builder.add_node("apply", nodes.apply_edits, defer=True)
    builder.add_node(
        "tester", nodes.tester,
        retry_policy=RetryPolicy(max_attempts=3, retry_on=Exception),
    )
    builder.add_node("fixer", nodes.fixer)
    builder.add_node("pr_approval", nodes.pr_approval)

    builder.add_edge(START, "intake")
    builder.add_edge("intake", "triage")
    builder.add_conditional_edges(
        "triage", nodes.route_by_type,
        {"bug": "planner", "feature": "planner", "chore": "planner"},
    )
    builder.add_edge("planner", "plan_approval")
    builder.add_conditional_edges("plan_approval", nodes.gate, ["editor", "abandon"])
    builder.add_edge("abandon", END)
    builder.add_edge("editor", "apply")
    builder.add_edge("apply", "tester")
    builder.add_conditional_edges(
        "tester", nodes.route_after_tests,
        {"done": "pr_approval", "give_up": END, "fixer": "fixer"},
    )
    builder.add_edge("fixer", "tester")
    builder.add_edge("pr_approval", END)
    return builder.compile(checkpointer=checkpointer, cache=InMemoryCache())

Two additions from M8: the cache_policy= on the planner node and the retry_policy= on the tester node. The compile(cache=InMemoryCache()) makes the cache live. The graph shape — nodes, edges, HITL interrupts — is identical to what you built in Module 7. Resilience is additive.

The durability mode is chosen at invoke time:

# For a run that modifies files and opens a PR — irreversible side effects
graph.invoke(
    {"issue": "...", "repo_path": repo},
    {"configurable": {"thread_id": "run-001"}},
    durability="sync",
)

# Resume after a crash
graph.invoke(
    None,
    {"configurable": {"thread_id": "run-001"}},
    durability="sync",
)

---

Hands-on lab: Build it

What you will build

This lab adds RetryPolicy on the Tester, CachePolicy on the Planner, InMemoryCache at compile time, and sandbox hardening — all on top of the Module 8 codebase. You will also run three demos: a flakiness simulation, a cache hit, and a path traversal block.

At the end, running the offline tests prints:

$ uv run pytest tests/ -q
tests/test_module_09.py::test_retry_policy_retries_until_success PASSED
tests/test_module_09.py::test_cache_policy_skips_recomputation PASSED
tests/test_module_09.py::test_sandbox_times_out_on_runaway_test PASSED
tests/test_module_09.py::test_durability_sync_completes PASSED
tests/test_pipeline.py::test_hitl_approve_plan_and_pr PASSED
tests/test_pipeline.py::test_retry_policy_recovers_transient_tester_failure PASSED
... N passed in X.XXs

No API key required. The pytest suite runs against the real Tally codebase — that is the observable, offline signal.

Step 1: Copy the Module 8 codebase

Start from the module-08/ directory byte-for-byte. Module 9 modifies exactly two files: forge/graph.py and forge/sandbox.py. Nothing else changes. No new fields on ForgeState, no new models, no new nodes.

Step 2: Update forge/graph.py

Add the three imports at the top:

from langgraph.cache.memory import InMemoryCache
from langgraph.types import CachePolicy, RetryPolicy

Then modify add_node for the planner and tester:

builder.add_node(
    "planner", nodes.planner,
    cache_policy=CachePolicy(key_func=lambda s: s.get("issue", ""), ttl=300),
)
# ...
builder.add_node(
    "tester", nodes.tester,
    retry_policy=RetryPolicy(max_attempts=3, retry_on=Exception),
)

And update compile:

return builder.compile(checkpointer=checkpointer, cache=InMemoryCache())

Step 3: Harden forge/sandbox.py

The current run_tests in Module 8 already has a timeout parameter (which you can verify in the lab code). What it lacks is:

• Explicit documentation of the timeout behavior (it already catches TimeoutExpired — good)
• A secret-filtering env in the subprocess call

Add the filtered env to run_tests:

def _filtered_env() -> dict:
    """Strip cloud credentials from the subprocess environment."""
    skip = ("ANTHROPIC_", "LANGSMITH_", "OPENAI_", "AWS_", "GOOGLE_", "HUGGING")
    return {k: v for k, v in os.environ.items() if not k.startswith(skip)}

And a safe write helper for path traversal prevention:

def write_file_safe(repo_path: str | Path, rel: str, content: str) -> str:
    """Write a file inside the sandbox only — raise ValueError on path traversal."""
    sb = Path(repo_path).resolve()
    target = (sb / rel).resolve()
    if not str(target).startswith(str(sb) + os.sep) and target != sb:
        raise ValueError(f"Path traversal blocked: {rel!r} escapes sandbox {sb}")
    target.parent.mkdir(parents=True, exist_ok=True)
    target.write_text(content)
    return str(target)

Step 4: Demo — flakiness simulation

Run this small script (or wire it into tests/test_pipeline.py with monkeypatch):

from unittest.mock import patch
from forge.graph import build_graph
from forge import checkpoint, sandbox

real_run = sandbox.run_tests
calls = [0]

def flaky_run(repo_path, timeout=120):
    calls[0] += 1
    if calls[0] == 1:
        raise RuntimeError("OSError: Text file busy — transient!")
    return real_run(repo_path, timeout)

with patch.object(sandbox, "run_tests", flaky_run):
    graph = build_graph(checkpointer=checkpoint.in_memory())
    # ... run the graph
    print(f"run_tests called {calls[0]} times")
    # Expected: 2 — failed once, retried once, succeeded, no Fixer

Step 5: Demo — cache hit

from forge.graph import build_graph
from forge import checkpoint

graph = build_graph(checkpointer=checkpoint.in_memory())
cfg1 = {"configurable": {"thread_id": "run-cache-1"}}
cfg2 = {"configurable": {"thread_id": "run-cache-2"}}

# First run — planner executes, result cached
graph.invoke({"issue": "same issue text", "repo_path": repo}, cfg1)
# Second run (same issue, different thread_id) — within ttl=300s, planner cache hit
graph.invoke({"issue": "same issue text", "repo_path": repo}, cfg2)
# The planner node ran once, not twice

Step 6: Demo — path traversal blocked

from forge.sandbox import write_file_safe

try:
    write_file_safe("/tmp/forge-test/tally", "../../etc/passwd", "hacked")
except ValueError as e:
    print(f"Blocked: {e}")
# Output: Blocked: Path traversal blocked: '../../etc/passwd' escapes sandbox ...

Try it yourself

• Lower max_attempts to 1 on the RetryPolicy. Run the flakiness demo. The first failure propagates to the graph, the Fixer activates, and the model call is wasted. Restore to 3.
• Set ttl=10 on CachePolicy. Wait 11 seconds between two runs of the same issue. Observe the Planner executing again. The TTL should reflect how long the repo contents are likely to stay stable.
• Switch to durability="exit" and simulate a crash by killing the process mid-run. On resume, Forge restarts from the beginning of the run (or the last interrupt() checkpoint). Switch to "sync" and repeat — resume picks up from the last completed node.

---

In production

Choosing the durability mode. For Forge specifically: "sync" is the right choice for any run that involves apply_edits or pr_approval. Both nodes have irreversible side effects — files written, PR opened. If the process crashes between those nodes and restarts from a stale checkpoint, Forge may apply the same edits twice or open duplicate PRs. The I/O overhead of "sync" is a worthwhile trade. Use "async" for exploratory or read-only runs (a Planner-only analysis, for example). Use "exit" for batch workloads where the entire run is cheap to redo and there is no human in the loop.

Calibrating RetryPolicy. In production, cap max_attempts at 2–3. Beyond that, a persistent failure is probably not transient — it is a configuration problem, a bad edit, or a broken dependency. Retrying 5 or 10 times wastes both time and tokens. Always set jitter=True in multi-threaded environments; without jitter, all threads that hit a transient failure will retry at the same moment, creating a thundering-herd effect that may overwhelm whatever service is failing. Use retry_on to narrow the retried exceptions: retry_on=OSError if you only expect subprocess I/O failures, not retry_on=Exception (which would silently retry programming errors like AttributeError).

CachePolicy TTL. The TTL should reflect the volatility of the data being cached. Tally’s source files change only when a developer pushes a commit, so a 5-minute TTL for the Planner analysis is reasonable for a single-developer setup. In a busy CI environment where many commits land per hour, you would lower the TTL or key the cache on the repo’s git HEAD hash rather than the issue text. A node that calls an LLM should never have a CachePolicy — LLM outputs are non-deterministic and often need fresh context.

Agentic security in production. Forge in OSS mode runs on your machine, which is already a relatively trusted environment. In LangGraph Platform (Module 14), the agent runs in a container — you get container-level process isolation, and you can add --network none to the pytest subprocess call or use network isolation at the container level. Regardless of environment: secrets must never be passed to the subprocess. The _filtered_env() pattern is not optional when your agent executes code. Think of it this way: running an agent that writes files without a sandbox is like running eval() on user input — you know you should not, but it is surprisingly easy to do by mistake. The path traversal check and the secret filter are one-liners that prevent an entire class of issues. Add them.

For self-hosted Forge: run the process under a dedicated system user with minimal filesystem permissions, not as root. Keep API keys out of the codebase and inject them at runtime through environment variables (or a secrets manager). Log every write_file call with the resolved path so you can audit what files the agent touched.

Module 15 will add LangSmith evaluation and a CI gate — at that point, security hardening becomes part of the eval checklist too.

---

Common misconception

“RetryPolicy and the Tester → Fixer loop do the same thing.”

They do not. They operate at completely different levels of the graph.

RetryPolicy retries the Tester node within a single superstep. The graph state does not change between retries. The attempts counter in ForgeState does not increment. No checkpoint is written between retries. The graph is not aware that any failure happened. This mechanism exists to absorb transient infrastructure failures: a subprocess race condition, a temporary file lock, a network timeout when reaching a remote test runner.

The Tester → Fixer loop spans multiple supersteps. Each iteration writes a checkpoint, increments attempts, and updates test_result in ForgeState. It exists to handle genuine test failures: the code is wrong, the edit did not fix the bug, the test logic needs to be revisited. The Fixer calls an LLM, produces new edits, and the Tester runs again on a different codebase state.

Combining them is correct and expected: the RetryPolicy absorbs transient subprocess errors before they reach the graph, and the Tester → Fixer loop handles true failures after a clean test run. They do not conflict because they operate at different granularities.

The second classic mistake in this module: writing checkpoint_during=True or checkpoint_during=False. That is the LangGraph v0.x API. The v1.x equivalent is durability="async" or durability="exit". The old parameter may still be accepted in some v1.x minor versions for backward compatibility, but its semantics may differ. All new code should use durability=.

---

Mastery corner

What to really understand here

The core insight of Module 9 is the hierarchy of durability mechanisms: durability= controls when state is persisted; RetryPolicy controls when a node is retried; CachePolicy controls when a node is skipped. They are orthogonal — you can combine any of them. The thread running through all three is idempotence: a node is safe to retry, cache, or replay only if running it twice with the same input produces the same output with no side effects. This is not a nice-to-have property; it is a prerequisite for production durability.

Sandboxing is in the same module because Forge executes code, and executing untrusted code is the point where “durable and resilient” intersects with “secure.” Path traversal, subprocess timeouts, and secret filtering are not advanced security topics — they are table stakes for any agent that writes files.

Questions

Q1. Forge has just displayed the plan approval interrupt and the user approved. The apply_edits node is about to write four files, and then pr_approval will open a PR. If the process crashes between apply_edits and pr_approval, what is the worst-case outcome with each durability mode?

A. "exit": restart from the beginning of the run; "async": probably restart from apply_edits; "sync": restart from exactly pr_approval (the apply_edits checkpoint was written before pr_approval started) B. All three modes restart from the last interrupt() (the plan approval), because interrupt() always forces a checkpoint C. "exit" and "async" both restart from the beginning of the run; "sync" restarts from apply_edits D. "sync" and "async" both restart from the last node boundary; "exit" always requires a full replay

Q2. A junior developer proposes adding retry_policy=RetryPolicy(max_attempts=5) to the Fixer node to make it “more robust.” What is the correct response?

A. This is fine — the Fixer is a critical node and should have the highest retry count B. This will retry the Fixer up to 5 times with the same state input, producing up to 5 different LLM-generated FileEdits that will all be applied, likely corrupting the codebase C. The RetryPolicy will only retry on exceptions; since the Fixer does not raise exceptions on bad output, it will have no effect D. This is correct but unnecessary because the Tester → Fixer loop already provides retries

Q3. You add CachePolicy(key_func=lambda s: s.get("issue", ""), ttl=300) to the Planner. A colleague pushes a new commit to the Tally repo 30 seconds after your first Forge run. You start a second Forge run on the same issue 60 seconds later. What does the Planner return?

A. The Planner re-executes because the TTL has not expired but the repo hash changed — LangGraph detects file modifications B. The Planner returns the cached Plan from the first run, because the cache key (the issue text) is the same and the TTL (300s) has not expired C. The Planner raises a CacheInvalidError because the repo content no longer matches the cached analysis D. The Planner re-executes because the InMemoryCache is invalidated whenever a new thread is started

Q4. Your sandbox contains /tmp/forge-abc123/tally/. A model generates FileEdit(path="../../secrets/.env", ...). You call write_file_safe("/tmp/forge-abc123/tally", "../../secrets/.env", content). What happens, and why does Path.resolve() matter here?

A. Path.resolve() is redundant — a string startswith check on the raw path would catch ../../ equally well B. The write proceeds because resolve() canonicalizes ../../secrets/.env relative to the sandbox root, producing a path that still starts with /tmp/ C. ValueError is raised: resolve() expands ../../secrets/.env to /tmp/secrets/.env, which does not start with /tmp/forge-abc123/tally/ — without resolve(), a symlink or a malformed .. chain might bypass a simple string check D. The write is blocked only if secrets is a directory that exists on disk — if it does not exist yet, the path traversal succeeds

Q5. The run_tests function in sandbox.py is called with timeout=120. A model-generated test file contains while True: pass. What happens, and why does the RetryPolicy on the Tester NOT help in this case?

A. subprocess.TimeoutExpired is raised after 120 seconds; RetryPolicy catches it and retries the subprocess, which hangs again — after max_attempts=3, the error propagates B. The subprocess hangs indefinitely because subprocess.TimeoutExpired is not a subclass of Exception and is not caught by retry_on=Exception C. subprocess.TimeoutExpired is raised after 120 seconds; the except block converts it to TestResult(passed=False, failures=["<timeout>"]); RetryPolicy would retry the node, but a timeout is not transient — the same code will hang again, so RetryPolicy wastes two more 120-second waits before giving up D. The subprocess hangs indefinitely — subprocess.run with timeout= only raises on network calls, not on CPU-bound infinite loops

Answers

A1: A. The interrupt() checkpoints are a special case — they always write a checkpoint regardless of the durability= mode. But between apply_edits and pr_approval, there is no interrupt(). With "exit", the run restarts from the post-interrupt checkpoint (plan approved state). With "async", the checkpoint for apply_edits was being written concurrently with pr_approval starting — it may or may not be complete, depending on the exact crash timing. With "sync", the checkpoint for apply_edits was guaranteed to be written before pr_approval started, so resume picks up at pr_approval. For irreversible side effects, "sync" is the correct choice.

A2: B. The Fixer is not idempotent. Each call to model.invoke() produces a different response for the same prompt. With max_attempts=5, a transient exception (e.g., an API timeout) would cause the Fixer to be called 5 times with the same ForgeState, producing 5 different FileEdits that are all applied sequentially in undefined order. The RetryPolicy was designed for idempotent nodes. Option D is wrong because the Tester → Fixer loop retries at the superstep level, with state updates, which is entirely different from retrying the same node invocation.

A3: B. The CachePolicy in this implementation keys on the issue text, not on repo content. The TTL is 300 seconds. 60 seconds into the window, the cache is still valid and the cached Plan is returned. This is a known limitation — if the repo changes between runs, a content-based cache key (e.g., the repo’s git HEAD hash) would be more accurate. For a solo developer where repo changes and Forge runs rarely overlap, the issue-text key with a moderate TTL is a reasonable trade-off.

A4: C. Path.resolve() follows all symlinks and collapses all .. components to produce the canonical absolute path. /tmp/forge-abc123/tally/../../secrets/.env resolves to /tmp/secrets/.env, which does not start with /tmp/forge-abc123/tally/. The ValueError is raised. Without resolve(), a raw str.startswith("/tmp/forge-abc123/tally") check would also catch this specific case — but a crafted path with a symlink inside the sandbox pointing outside, or a path like /tmp/forge-abc123/tally/../../../etc/passwd, might bypass a non-canonicalized check. resolve() is the safe, correct approach.

A5: C. The sandbox.run_tests function catches subprocess.TimeoutExpired and converts it to a clean TestResult(passed=False). That is correct behavior. However, the RetryPolicy would then retry the node — and the same code would hang for another 120 seconds on each retry, for a total wait of max_attempts * timeout = 3 * 120 = 360 seconds. A timeout is not a transient error: the infinite loop does not fix itself between retries. In production, you would use retry_on to exclude subprocess.TimeoutExpired from the retry set: retry_on=lambda e: not isinstance(e, subprocess.TimeoutExpired).

Traps to avoid

Trap 1 (v0 → v1 freshness): Using checkpoint_during= in new code. checkpoint_during=True is the LangGraph v0.x parameter for what is now durability="async". In v1.x code, write graph.invoke(input, config, durability="async"). The old parameter may still compile without error in some 1.x patch versions due to backward compatibility shims, but the semantics may not match — and future versions will drop it. Every checkpoint_during= in new code is a bug waiting to be triggered on upgrade.

Trap 2: Adding RetryPolicy to a non-idempotent node. Putting RetryPolicy on the Fixer, an editor, or any other node that calls a language model is a cost amplifier, not a safety net. Each retry produces a different output. If the node has side effects (writing to disk, calling an API), those side effects multiply with max_attempts. Only attach RetryPolicy to nodes that are genuinely idempotent: deterministic computations, subprocess invocations on unchanged inputs, read-only API calls.

Trap 3: Forgetting compile(cache=...) after adding CachePolicy. This one is silent. You add cache_policy=CachePolicy(ttl=300) to a node, run the agent, and wonder why the Planner is still being called on every run. The answer is that compile(cache=InMemoryCache()) was not passed. Without the cache backend at compile time, all CachePolicy declarations are ignored. No exception, no warning, no log line. The fix is a single argument in builder.compile().

---

Key takeaways

• durability= on invoke / stream controls the checkpoint timing: "exit" (end of run), "async" (default, overlapping with next node), "sync" (before next node). Never write checkpoint_during= in new LangGraph v1.x code — that is the deprecated v0.x form.
• RetryPolicy absorbs transient failures within a single superstep, invisibly to the graph. It is not the same as the Tester → Fixer loop, which spans multiple supersteps and updates state. Only attach it to idempotent nodes.
• CachePolicy + InMemoryCache skip node execution when the cache key matches within the TTL. The compile(cache=...) argument is required — without it, cache policies are silently inactive.
• @task in the Functional API is the unit of durable replay: completed tasks are replayed from the checkpoint on resume rather than re-executed. It accepts the same retry_policy= and cache_policy= primitives. The full Functional API build is Module 13.
• Forge executes model-generated code — that makes it an attack surface. Path traversal (resolve() + prefix check), subprocess timeout, and secret filtering in the subprocess environment are production requirements, not optional hardening.
• Secrets (ANTHROPIC_API_KEY, LANGSMITH_API_KEY, cloud credentials) must never appear in the subprocess environment. Filter them explicitly before every subprocess.run.
• The three storage primitives are distinct: InMemorySaver / SqliteSaver (checkpointer, persists state between runs), InMemoryCache (node cache, memoizes outputs by key + TTL), and the Store (long-term cross-thread memory, Module 10).

---

What’s next

Forge can now retry a flaky test without wasting a fix loop — but it still forgets everything between runs. Module 10 gives it long-term memory via the Store: Tally conventions, past fixes, and semantic recall so the Planner knows what worked last time.

---

Want the full LangGraph mastery question bank (interview-style) and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

This is Module 09 of LangGraph in Production, a free 16-module course that takes you from your first StateGraph to a deployed, durable autonomous coding agent on LangGraph v1.x. Start at Module 1 or browse the full syllabus.

Links:

• Lab code — module-09
• Previous module — M08: Time-Travel, State History, and Forking
• Next module — M10: Long-Term Memory and the Store
• Course index

---

References

1. LangGraph documentation — Durability: https://langchain-ai.github.io/langgraph/concepts/durability/ (as of June 2026)
2. LangGraph API reference — RetryPolicy and CachePolicy from langgraph.types: https://langchain-ai.github.io/langgraph/reference/types/ (as of June 2026)
3. LangGraph API reference — InMemoryCache from langgraph.cache.memory: https://langchain-ai.github.io/langgraph/reference/cache/ (as of June 2026)
4. LangGraph how-to — Add node retry policies: https://langchain-ai.github.io/langgraph/how-tos/node-retries/ (as of June 2026)
5. LangGraph how-to — Add node caching: https://langchain-ai.github.io/langgraph/how-tos/node-caching/ (as of June 2026)
6. Python documentation — subprocess module, TimeoutExpired: https://docs.python.org/3/library/subprocess.html#subprocess.TimeoutExpired (Python 3.12)