aifromzero.dev

The FM Integration Layer: Model Selection, Routing, Streaming, and Resilience (AWS GenAI Developer Pro, Module 3)

Module 3 of 16 15 min read D1 · 31%D2 · 26% Lab code ↗

This is Module 3 of AWS GenAI Pro Mastery, a free 16-module course that takes you from your first Amazon Bedrock call to AWS-certified.

It is the Monday of a sale week at CloudCart. Tickets land at ~3,000 an hour, and Relay’s Module 2 triage starts taking ThrottlingException in bursts. The retry someone wrote in a hurry is an immediate loop — it fires again the millisecond a call fails, so every throttled request becomes three, and the storm feeds itself. Meanwhile every ticket, even “where is my order?”, pays for the exact model that handles a billing dispute, because the model ID is hardcoded one layer up. The triage “worked” in the demo; under load it is a cost leak wrapped in an outage.

Calling a foundation model is not integrating one. This module builds Relay’s FM integration layer: a design doc that pins the architecture, then relay/llm.py — a single converse() that routes between a fast and a smart tier by request complexity, streams, absorbs throttling with backoff, and falls back across Regions. By the end, no file except relay/config.py knows a model ID exists.

In this module

  • You’ll learn to select a foundation model with evidence, architect a model switch with no code change, build resilience (backoff, cross-Region inference, graceful degradation), stream responses and know when it pays, route versus cascade, and explain the custom-model lifecycle.
  • You’ll build Relay’s design doc and relay/llm.py: one converse() with fast/smart routing, streaming, retries, and cross-Region fallback — the LLM contract for the rest of the course.
  • Exam domains covered: D1 — Foundation Model Integration, Data Management, and Compliance — 31% (this module owns Task 1.2 end to end) and D2 — Implementation and Integration — 26% (skills 2.4.2/2.4.3/2.4.4, plus 2.2.3 in part) — the single largest pool of D1+D2 questions in the course.
  • Prerequisites: Module 1 (a secured account with AWS_PROFILE, a working Converse call) and Module 2 (the relay/ package, the Ticket/Triage schemas, governed prompts, and the test suite). Region is us-east-1 throughout.

Where you are in the build

You are at Module 3 of 16. The destination is the AWS Certified Generative AI Developer - Professional (AIP-C01) exam, and the vehicle is Relay, CloudCart’s support agent.

  • ✅ M1 — secured account, budget alarm, first Converse call.
  • ✅ M2 — triage: structured output + Prompt Management + a prompt test suite.
  • 👉 M3 — the FM integration layer: model routing, streaming, retries, cross-Region fallback.
  • ⬜ M4–M16 — RAG, agents, safety, deployment, cost, evals, observability, capstone.

Relay before this module is a triage step that calls one hardcoded model with no network retry and no streaming — a demo, not a system. After it, Relay has a design doc, a routed and resilient LLM layer, and a triage refactored to call llm.converse(tier="fast") — its 10/10 suite re-passing, untouched. This is a freeze event: the converse() signature you write today is byte-identical through Module 15.

Write the design doc first

Before any code, you write docs/relay-design.md. Writing it first is the point: it is where you decide, on paper, that triage is fast-tier work, that a billing dispute is smart-tier, that a top-end model exists only as a reference, and that one API talks to every model. The code then implements a decision you already made.

The exam calls this skill 1.1.1, solution design: you translate requirements into an architecture before you build, and a one-page design doc with dated decisions is the smallest honest version. Relay’s records four:

  • Two production tiers, fast + smart. Routing is a cost decision before an architecture decision; paying a reasoning model for “where is my order?” is waste.
  • Converse / ConverseStream everywhere, never the legacy single-prompt invoke path. One API across every text model gives one message shape, one streaming path, and one place to bolt on tool calling and guardrails later. (The course-wide exception is Titan embeddings in Module 4.)
  • Inference profiles, never bare regional IDs — more below; it is the exam’s favorite trap.
  • us-east-1 for the whole course — one Region keeps IAM, quotas, and pricing simple; cross-Region behavior is handled by the profiles, not a multi-Region application.

The doc also draws the layer: everything in the top box exists after this module; the bottom box is forward reference, drawn so the seam is clear, not because it is built.

graph TD
    subgraph Built by Module 3
        T["triage.py / future callers"] --> C["llm.converse()"]
        C --> R{"router (tier=auto)"}
        R -->|fast| PF["Nova Micro profile"]
        R -->|smart| PS["Nova 2 Lite profile"]
        C --> RB["retry: exponential backoff + jitter"]
        RB --> FB["fallback: alternate cross-Region profile, then degrade tier"]
        PF -.->|IDs live ONLY in| CFG["relay/config.py"]
        PS -.->|IDs live ONLY in| CFG
    end
    subgraph Planned later — NOT built yet
        INTAKE["intake.py: validate / vision / PII (M6, M10)"]
        AGENT["Strands agent + tools + MCP (M7-M8)"]
        KB["Knowledge Base relay-kb (M5)"]
        API["API Gateway + Lambda + SQS (M11)"]
    end
    API -.-> INTAKE -.-> AGENT
    AGENT -.-> C
    AGENT -.-> KB

The rule for the whole course: nothing past the current module is described as built. The agent, the Knowledge Base, the API each arrive in their own module — and every one calls llm.converse() without ever naming a model ID.

Choosing models with evidence

Skill 1.2.1 wants a method for picking a foundation model, not a vibe — three filters, in order.

First, public benchmarks as a coarse filter. Leaderboards tell you which models are roughly in contention — not which one is right, because they run on someone else’s data, not yours.

Second, capability analysis against your actual constraints: context window, modalities (Module 6 needs vision; triage does not), languages (CloudCart gets French and English tickets), latency class, and price per million tokens. This is where most models drop out — not for scoring badly, but for costing too much or lacking a feature you need.

Third — the only filter that actually decides — testing on your own data. Relay already has a benchmark: the 10 reference tickets and deterministic suite from Module 2. A model that scores 10/10 on those at temperature 0, cheaply, is the fast tier. You adopt a model because it passes your suite, not a leaderboard.

Run that method and Relay lands on two production tiers plus one reference point. Prices are stamped as of June 2026 — re-verify on the Bedrock pricing page — and all three IDs are us. inference profiles, never bare regional IDs.

TierModel (inference profile)Price in / out ($/M tok)ContextRelative latencyStrengths / limitsRole in Relay
fastNova Micro (us.amazon.nova-micro-v1:0)~$0.035 / $0.14text, largelowestCheapest capable text model; classification and short replies. Not for multi-step reasoning.Triage, the router’s own floor, tests
smartNova 2 Lite (us.amazon.nova-2-lite-v1:0)~$0.30 / $2.50up to 1MmediumTunable reasoning, very large context. ~8x the input cost of fast.Complex answers, billing disputes, agent reasoning
frontierClaude Sonnet 4.5 (us.anthropic.claude-sonnet-4-5-20250929-v1:0)higher (re-verify)largehighestCeiling-grade reasoning and wording. Overkill for support triage.Reference / Try-it-yourself only — a yardstick, never default traffic

The opinion I’ll defend: 80% of CloudCart tickets are fast-tier work — “where is my order?”, “how do I reset my password?” — short, single-step, answerable by the cheapest capable model. Routing is a cost decision before an architecture decision: send all of that to the smart tier “to be safe” and you have multiplied your input-token bill by roughly eight for no quality the customer can perceive. The frontier tier exists so you can measure that the expensive model is not worth it.

Routing and cascading: three ways to pick a model at runtime

How do you choose a tier per request? The exam (skills 2.4.4 and 2.2.3) wants you to tell four strategies apart.

StrategyPrincipleLatencyCostComplexityWhen the exam picks it
Static routing (by config)One model for the whole workload; change it in configbaselinefixedlowestA single homogeneous task; you just want to swap models without a redeploy
Routing by content/complexity (Relay’s choice)Classify the request, send it to the right tier oncebaseline (one classify step)low — most traffic on the cheap tierlowMixed traffic, clear cheap-vs-expensive split, tight budget
Routing by metricsPick the tier from live latency/error telemetrybaseline + telemetry readvariablehighYou need to shed load or dodge a degraded model in real time
Model cascadingAlways try the small model; escalate to a bigger one on a poor/failed resulthigher on hard cases (two calls)low on easy cases, higher on hard onesmediumQuality floor matters more than worst-case latency; you can judge “good enough” cheaply

Relay routes by content/complexity: the router reads the request, classifies it, and sends it to one tier — one call. Model cascading does the opposite — it always calls the small model first, then makes a second call to a bigger one when the first result looks wrong, buying a quality floor at the price of doubled latency on the hard cases. Relay does not cascade; on a billing dispute it pays for the smart tier once, up front. Routing by metrics (a tier chosen from live latency/error telemetry) is named here but stays theory: the router reads the request, not a metrics feed.

⚠️ Common misconception: routing and model cascading are the same thing. They are not, and the exam loves the gap. Routing classifies the request and sends it to the right model once. Cascading always tries the small model and escalates on a poor result — a second call on failure. Different cost curves (routing is flat; cascading is cheap on easy cases, expensive on hard ones) and different latency. “Try small, fall back to large” is cascading; “look at the request, pick the model” is routing.

The switch with no redeploy

Skill 1.2.2 is about changing the model in production without touching code. Relay’s first answer is the containment law: every model ID lives in relay/config.py, so changing the smart tier is a one-line config edit plus a restart.

The production-grade answer externalizes that map entirely. AWS AppConfig is the managed configuration service: you store the tier→profile map as a freeform configuration, your code loads it at startup, and you change the model in AppConfig and roll it out — with validators and a deployment strategy — without redeploying the application. That is the lab’s “Try it yourself,” and the exact shape the exam means by “model selection is configuration, not code.” A code change and redeploy is the wrong answer.

Streaming and resilience: ConverseStream, backoff, cross-Region

This half of the module keeps Relay up under load: deliver the answer well, and survive the failures.

ConverseStream and when streaming matters

ConverseStream is the streaming sibling of the Converse API: it sends a sequence of events, and you read text deltas (contentBlockDelta) as the model produces them. The number that matters is time-to-first-token — how long the human waits before anything appears. Total latency is unchanged; perceived latency drops sharply, because the answer starts scrolling immediately.

So stream for long answers in front of a human (skill 2.4.2). Do not stream for a parser: triage returns a one-line JSON object you validate before acting, so you need the whole object first and streaming buys nothing. That is why triage runs stream=False and the demo streams.

Resilience: backoff, jitter, cross-Region, graceful degradation

When a call fails, how you retry decides whether you recover or make it worse. The fix is exponential backoff with jitter (skill 2.4.3): grow the ceiling after each failed attempt — Relay’s two retries cap at ~1s then ~2s — and sleep a random duration between 0 and that ceiling (full jitter) so a fleet of clients does not retry in lockstep, the thundering herd that turns one throttle into a storm.

Two more levers stack on top. Cross-Region inference: when one profile is throttled past its retries, fall back to the tier’s alternate global. profile, drawn from a wider Region pool. Then graceful degradation as a last resort: a smart call that cannot get through anywhere drops to fast and still answers. Past two network retries, Relay degrades or fails — a third retry on a broken call is just latency you bill the user for.

Name one heavier pattern too: a circuit breaker trips open after repeated failures and stops calling a dead dependency for a cooldown; the canonical AWS example orchestrates it with Step Functions. Relay does not build one — backoff-then-degrade is enough at this scale — but recognize the pattern.

This is the full failure path, in order:

sequenceDiagram
    participant Caller
    participant LLM as llm.converse()
    participant Primary as primary profile (us.)
    participant Alt as alternate profile (global.)
    Caller->>LLM: converse(messages, tier="smart")
    LLM->>Primary: Converse
    Primary-->>LLM: ThrottlingException
    Note over LLM: backoff + jitter (attempt 1, 2)
    LLM->>Primary: Converse (retry)
    Primary-->>LLM: ThrottlingException
    Note over LLM: primary exhausted → cross-Region fallback
    LLM->>Alt: Converse (alternate profile)
    Alt-->>LLM: ThrottlingException
    Note over LLM: all profiles exhausted → degrade smart → fast
    LLM->>LLM: retry chain on fast tier
    LLM-->>Caller: answer (degraded tier) — or LLMError, never silent

⚠️ Common misconception: “Cross-Region inference is a disaster-recovery feature I turn on when a Region goes down.” Wrong — it is the nominal mode for recent models. The model ID you call is an inference profile (us. or global.), and that profile routes requests across the Regions in its geography for capacity, on every call. Per the AWS docs the geographic us. profiles add no routing surcharge over the base on-demand price, and a global. profile can carry a different (often lower) per-token rate — re-verify the exact delta on the pricing page as of June 2026 before you rely on it. A bare regional ID (e.g. amazon.nova-micro-v1:0) invoked on-demand fails with “Retry with an inference profile.” Cross-Region is not a failover you flip during an outage; it is how these models are meant to be called. (DR is a separate, application-level concern.)

Custom models: the lifecycle you must know (and won’t build)

Skill 1.2.4 is on the blueprint, so you need to explain model customization even though Relay never trains a model. For a support agent, customization is the wrong first reach: when the model lacks knowledge, you give it knowledge with RAG, not new weights (Module 4). But the lifecycle is exam material, so here is the shape.

You start from a foundation model and fine-tune it — most efficiently with a parameter-efficient method like LoRA (Low-Rank Adaptation), which trains a small set of adapter weights instead of the whole model. The trained artifact is a versioned asset: you register it in the Model Registry (the SageMaker AI Model Registry is the canonical home), where each version carries its metadata, approval status, and lineage. A deployment pipeline promotes an approved version to an endpoint; if it regresses, you roll back to the previous registered version. The loop the exam wants you to recognize: fine-tune (LoRA) → register and version → deploy via pipeline → monitor → roll back or retire. Relay reaches for none of it — its gap is knowledge, not behavior.

Build it: Relay’s FM integration layer

Goal: write the design doc, then build and freeze relay/llm.py — routing fast/smart, streaming, retries, cross-Region fallback — and refactor Module 2’s triage to call it. Observable result: one command shows the router’s decision, a streamed answer, and a cost line; another shows the fast path; an offline test proves backoff. This lab cost me $0.01 on June 2026 prices — measured at $0.0065 from the Converse usage blocks, most of it (~75%) the optional frontier call ($0.0049); fast-tier traffic including the 13k-token triage suite is ~$0.0005. Full code lives in module-03/; excerpts below.

Step 1 — relay/config.py: the only place a model ID lives

relay/config.py is new here and is the sole home of model-ID literals in the whole repo. It maps a Relay tier to an inference profile; nothing else in relay/ may name a us./global. ID.

TIERS = {
    "fast":     "us.amazon.nova-micro-v1:0",   # triage, router floor, tests
    "smart":    "us.amazon.nova-2-lite-v1:0",  # complex answers, agent reasoning
    "frontier": "us.anthropic.claude-sonnet-4-5-20250929-v1:0",  # reference only
}
ALT_PROFILES = {"smart": "global.amazon.nova-2-lite-v1:0"}  # wider Region pool
DEFAULT_TIER = "fast"   # the router's floor: ~80% of tickets are fast-tier work

A grep gate proves nothing leaks out of this file, and it runs in the test suite:

grep -rE '(us|global|eu)\.(amazon|anthropic)\.' relay/ | grep -v config.py
# -> empty

The file grows by addition later, but the tier map itself is never re-pointed after this module.

Step 2 — relay/llm.py: the frozen converse()

This is the load-bearing file of the course. The signature is frozen here, byte-identical through Module 15:

converse(messages, *, tier="auto", stream=False, **params)

It is the unique Bedrock call site for all Relay code. The body has three jobs — route, stream, survive — and resilience is the heart of it. We disable botocore’s own retries so the backoff is explicit and teachable, not delegated to the SDK:

_MAX_RETRIES = 2                      # past 2, degrade or fail — never spin
_BACKOFF_BASE, _BACKOFF_CAP = 0.5, 8.0
_RETRYABLE_CODES = frozenset({"ThrottlingException", "TooManyRequestsException",
                              "ServiceUnavailableException", "ModelTimeoutException",
                              "InternalServerException", "ModelNotReadyException"})

def _backoff_sleep(attempt: int) -> None:
    ceiling = min(_BACKOFF_CAP, _BACKOFF_BASE * (2 ** attempt))
    time.sleep(random.uniform(0, ceiling))   # full jitter: no lockstep retries

The retry/fallback loop is the same for streaming and non-streaming — only the inner call differs. It tries each profile with backoff, then the alternate cross-Region profile, then degrades the tier, raising LLMError when every avenue is exhausted. No silent try/except: a non-retryable error (validation, access denied) surfaces immediately. One scope limit for streaming: the loop wraps opening the stream, so a fault that lands mid-stream — after the first deltas arrive — surfaces to the caller rather than being retried (a deliberate lab-scale simplification).

def _with_resilience(invoke, *, tier):
    tried_tier, last_error = tier, None
    while tried_tier is not None:
        for profile in _profile_chain(tried_tier):      # primary, then global. alt
            for attempt in range(_MAX_RETRIES + 1):
                try:
                    return invoke(profile), tried_tier
                except ClientError as err:
                    code = err.response["Error"]["Code"]
                    last_error = err
                    if code not in _RETRYABLE_CODES:
                        raise LLMError(...) from err     # surface the real cause
                    if attempt < _MAX_RETRIES:
                        _backoff_sleep(attempt + 1)
        tried_tier = _degrade(tried_tier)                # smart -> fast, then None
    raise LLMError("exhausted retries, fallback, and degradation", cause=last_error)

Step 3 — the complexity router (tier="auto")

The router is deliberately simple and explainable: a keyword/length heuristic, floored at fast, escalating to smart only on a reasoning signal. It is pure and deterministic — the suite checks it on fixed inputs — and it returns why:

route([{"role": "user", "content": [{"text": "hi"}]}])
# RouteDecision(tier='fast', reason='no complexity signal — default fast tier')

route([{"role": "user", "content": [{"text": "Why was I charged twice for order #1042?"}]}])
# RouteDecision(tier='smart', reason="matched complexity keyword 'charged twice'")

Step 4 — refactor triage through the layer

relay/triage.py is the only inherited file that changes, and only to route through the layer: it drops its provisional Module 2 model constant and calls converse(tier="fast"), inheriting backoff and cross-Region fallback for free. The validation flow is unchanged:

_TIER = "fast"   # triage IS the fast workload; nothing to route — so not "auto"

result = llm.converse(
    [{"role": "user", "content": [{"text": prompt_text}]}],
    tier=_TIER,
    inferenceConfig={"maxTokens": 100, "temperature": 0.0},
)

Prove non-regression — the Module 2 suite must still pass 10/10:

uv run python setup.py            # pings fast+smart, ensures the relay-triage prompt
uv run python run_prompt_tests.py # the inherited 10-ticket regression suite
score: 10/10
tokens: in=13092 out=220 | est. cost: $0.000489

Same 10/10, same token counts as Module 2 — refactoring the call path changed nothing the customer sees.

Step 5 — see it route, stream, and cost a call

uv run python demo_llm.py "Why was I charged twice for order #1042?"
router: tier=smart, reason=matched complexity keyword 'charged twice'

response (streaming):

I'm sorry for the double charge on order #1042. Here's what I'd check ...
[streams token by token]

tokens: in=78 out=126 | tier=smart | est. cost: $0.000338

And the fast path, no streaming:

uv run python demo_llm.py "hi" --no-stream
router: tier=fast, reason=no complexity signal — default fast tier

response:

Hi! How can I help with your CloudCart store today?

tokens: in=41 out=18 | tier=fast | est. cost: $0.000004

Step 6 — prove the resilience path offline

You do not need a real outage to see the backoff. The offline suite stubs a ThrottlingException and asserts the layer retries with backoff, and that a smart call throttled on both its primary and global. profiles degrades to fast and answers:

uv run python -m pytest tests/ -k retry

The full offline suite is 29 passed, 2 skipped (the skips are the live tests), guarding the frozen converse() signature byte-for-byte, the canonical tiers, the grep gate, the router, backoff-on-throttling, degrade-to-fast, the streaming delta path, and the routed triage flow.

Try it yourself. (1) Switch the model with no redeploy. Deport the tier map into an AWS AppConfig freeform configuration (relay-model-config), load it in config.py at startup with the in-code map as fallback, then change the smart tier in AppConfig and restart — no code change. (teardown.py deletes the app for you.) (2) Add a frontier tier and measure the delta. It is already in config.py; run three tricky tickets through --tier smart and --tier frontier and compare cost lines. That one frontier call was ~75% of this lab’s cost, and for CloudCart triage the answer is almost never worth ~10x the price — which is why it is a reference tier. Then uv run python teardown.py; the $5 budget stays.

In production

At one ticket a second, three things change. First, quotas are per model and per Region. Each inference profile has its own on-demand request and token quotas (Service Quotas); the fast and smart tiers throttle independently, so a “sudden” outage is usually one tier hitting its ceiling while the other is fine. You request increases before the sale.

Second, the router becomes a policy, not a function. “Who gets the smart tier?” is a team decision with a budget attached, and the router’s reason string is your audit trail: every escalation carries a recorded justification. Those routing metrics feed a dashboard — you’ll wire them into CloudWatch in Module 14.

Third, two cost levers wait in the wings: provisioned throughput buys guaranteed capacity at a fixed hourly rate for large, steady workloads (a Module 11 decision), and a Flex service tier trades latency for a discount on jobs that can wait (priced in Module 12). At lab scale, on-demand inference profiles are correct and idle-bill nothing.

Exam corner

What the exam tests here. This module owns D1’s Task 1.2 end to end (1.2.1 select FMs, 1.2.2 switch without code change, 1.2.3 resilient systems, 1.2.4 customization lifecycle) and a large slice of D2’s Task 2.4 (2.4.2 streaming, 2.4.3 reliable FM calls, 2.4.4 intelligent routing), plus 2.2.3 (small models / cascading) — the densest D1+D2 pool in the course.

Five scenario questions in the exam’s style; answers follow.

1. Your team must change the foundation model a production service uses — with no code change and no redeploy — and roll it back if it regresses. Which fits best? A. Hardcode the model ID and ship a hotfix when it changes. B. Store the tier→model map in AWS AppConfig as a freeform configuration the app loads at startup, and deploy changes through AppConfig. C. Set the model ID in a Lambda environment variable and redeploy for every change. D. Put the model ID in the prompt text in Prompt Management.

2. During a spike, a service gets ThrottlingException in bursts. The code retries immediately in a tight loop, and it gets worse. The correct fix? A. Remove the retries so failed calls fail fast. B. Retry with exponential backoff and jitter, capped at a small number of attempts. C. Raise maxTokens so each call does more work. D. Retry immediately but on a second thread.

3. A call to a recently released model fails telling you to “retry with an inference profile,” though the model is available in your Region. Cause and fix? A. The model is not enabled; submit a model-access request. B. You called a bare regional model ID on-demand; call it through its us./global. cross-Region inference profile instead. C. Your IAM role lacks bedrock:InvokeModel; add it. D. The Region is wrong; switch to us-west-2.

4. Users abandon a chat assistant because long answers take seconds to appear. A separate batch job classifies tickets into a JSON object you parse. Where should you add streaming? A. Both — streaming always reduces latency. B. The chat assistant only — stream long human-facing answers via ConverseStream; the JSON classifier needs the whole object before it can act. C. The classifier only — streaming makes parsing faster. D. Neither — streaming is a UI gimmick.

5. 80% of your requests are simple, 20% are complex, and the budget is tight. You want low cost without sacrificing quality on the hard cases. Which design fits best? A. Send everything to the frontier model to be safe. B. Route by complexity: classify each request and send simple ones to a cheap tier, complex ones to a stronger tier, in a single call. C. Always call the cheap model and escalate to the frontier model only when the result fails (cascading), accepting the extra latency. D. Send everything to the cheap model and accept the quality drop on the 20%.

Answers. 1 → B. Model selection is configuration, not code: AppConfig externalizes the map and supports rollout/rollback with no redeploy; A and C still require a deploy. 2 → B. Exponential backoff + jitter, capped, is the textbook 2.4.3 answer; the immediate loop is the anti-pattern, and maxTokens (C) is unrelated. 3 → B. Recent models require an inference profile on-demand; the bare regional ID is the trap and the us./global. profile is the fix. The manual console model-access step disappeared in late 2025 for serverless FMs (Anthropic still needs a one-time use-case form), but A is wrong here regardless: the error is a profile error, not an access denial. 4 → B. Stream long human-facing answers; a parser needs the whole object, so streaming a JSON classifier helps nothing. 5 → B. Routing by complexity gives flat, low cost with quality on the hard cases in one call; C (cascading) works but pays doubled latency on every hard case.

Traps to avoid.

  • Confusing cross-Region inference (the nominal mode of us./global. profiles, routing for capacity on every call) with a multi-Region application failover you flip during an outage. The profile IS the model ID.
  • Retrying without backoff or jitter — an immediate or lockstep retry is the thundering herd that amplifies the throttle it is meant to absorb.
  • Confusing routing (classify, send once) with model cascading (try small, escalate on failure — a second call). Different cost and latency; the exam tests it directly.

Key takeaways

  • There is exactly one Bedrock call site in Relay’s code — relay/llm.py’s converse(), frozen here through Module 15. Every later file calls it; none names a model ID.
  • The modern model ID is an inference profile (us./global.), not a bare regional ID. Cross-Region inference is the nominal mode — it routes for capacity on every call — not a DR toggle.
  • Retry is exponential backoff + jitter, capped, never an immediate loop. A lockstep retry amplifies throttling into an outage.
  • Routing by complexity captures ~80% of the savings — most tickets are fast-tier work, and routing is a cost decision before an architecture decision.
  • Stream for humans, not for parsers. ConverseStream cuts time-to-first-token on long answers; a JSON classifier needs the whole object.
  • Routing ≠ cascading. Routing classifies and sends once; cascading tries small and escalates on failure — different cost and latency.
  • The custom-model lifecycle is exam knowledge, not a build: fine-tune (LoRA) → Model Registry → deploy pipeline → roll back/retire. Relay reaches for RAG instead.

What’s next

Relay can talk — but it knows nothing about CloudCart. Module 4 starts RAG: chunking the help-center docs, embedding them with Titan into 1024-dimensional vectors, and picking a vector store (S3 Vectors) that doesn’t bill you while you sleep. The converse() contract you froze today is exactly what the grounded answer path will call.

Want the full AIP-C01 question bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Module 3 lab · ← Module 2 · Course index · Module 4 →

References