Evaluating GenAI Applications: Bedrock Evaluations, LLM-as-a-Judge, and RAG Metrics (AWS GenAI Developer Pro, Module 13)

This is Module 13 of AWS GenAI Pro Mastery, a free 16-module course that takes you from your first Amazon Bedrock call to AWS-certified.

Last module you compressed Relay’s prompts and shifted triage onto the cheapest model to cut the bill 40%. This morning a customer complains that Relay answered a refund question with a confident, plausible, completely wrong reply. Did your cost optimization break answer quality, or is this one unlucky ticket? You go to find out — and discover there is nothing to find out with. You reread five tickets by hand, they “look fine,” and you ship the fix on a hunch. That is the whole problem: a GenAI system without evaluation cannot be debugged, only guessed at. Deployed, governed, and costed, Relay has never once produced an objective number for “is it good?” By the end of this module, a single number answers “did we just make Relay worse?” — and the deployment pipeline blocks itself when the answer is yes.

In this module

You’ll learn how to:

• Design an evaluation framework for foundation-model output that goes beyond classic ML metrics — relevance, factual accuracy, consistency, fluency.
• Build a 20-ticket golden dataset and run reproducible evals on it, with aggregate score reporting.
• Run a Bedrock RAG evaluation job on Relay’s Knowledge Base — correctness, faithfulness, completeness, citation precision.
• Implement a custom LLM-as-a-judge for triage and the agent — task completion, tool usage — while avoiding self-preference bias.
• Gate deployments: continuous evaluation plus a regression gate wired into the Module 11 CodePipeline.
• Apply a fairness evaluation via a judge (skill 3.4.2) and collect user feedback in a loop.

You’ll build: Relay’s eval harness — a 20-ticket golden set, a Bedrock RAG evaluation job, a custom LLM-as-a-judge, and a regression gate wired into the Module 11 pipeline.

Exam domains covered: D5 — Testing, Validation, and Troubleshooting — 11% (this module covers Task 5.1, evaluation; Module 14 covers 5.2, troubleshooting). D3 — one skill: fairness evaluation (3.4.2), shared with Module 10.

Prerequisites: Modules 1–12, an AWS_PROFILE configured for us-east-1.

Where you are in the build

You are 13 modules into a 16-module build. Relay — CloudCart’s customer-support agent — is deployed, governed, and cost-optimized, but its quality has never been measured:

• ✅ M1–M12 — Relay triages, answers from a Knowledge Base with citations, acts through tools and an agent, is guarded and PII-redacted, ships behind an API with CI/CD, and knows what every ticket costs.
• 👉 M13 — you are here. A committed golden set, baselined scores, and a regression gate in the pipeline.
• ⬜ M14–M16 — observability, the capstone, and the exam.

Relay after this module: a versioned 20-ticket golden set, a baselined grounding number, a custom judge, a managed RAG evaluation that agrees with it, and a pipeline that refuses to deploy a regression.

Why evaluating a foundation model is not like evaluating ML

Classic ML evaluation has a luxury you have lost: a single correct label. A spam classifier’s output is spam or not-spam, you diff it against ground truth, and accuracy falls out. A good support answer has fifty valid phrasings. “Refund the duplicate charge from Billing → Transactions; it lands in 5–10 business days” and “Head to Billing → Transactions, refund the later charge, expect the money within a week or two” are both correct, yet a string match scores both as misses against any reference you wrote. There is no single label to diff against — so you cannot reuse a classifier’s accuracy metric, and the exam evaluates on different axes.

Per the official AIP-C01 exam guide, you score a foundation model’s output on four qualities. They are relevance (does the answer address what the customer actually asked?), factual accuracy (are the claims true and supported, with no invented promises?), consistency (does the same input produce a stable answer run to run?), and fluency (is it clear, well-formed, professional?). These are the FM-evaluation axes — none of them is “accuracy against a label.” You will score relevance and factual accuracy directly; you will get consistency for free by judging at temperature 0; fluency you mostly read off the tone.

The golden dataset

The instrument that makes all of this reproducible is the golden dataset: a curated, versioned set of inputs with what a good response must contain. Relay’s is 20 CloudCart tickets, and the mix is deliberate. There are 12 nominal cases (the bread-and-butter billing, technical, account, and shipping questions) and 4 edge cases (an empty message, a furious bilingual customer, an all-caps wall of text, a multi-question pile-up). The last six stress the hard paths: 2 adversarial cases (the prompt-injection and jailbreak family you built in Module 9) and 2 multimodal cases (a Module 6 checkout screenshot). Twenty is enough to start: it covers your real failure modes, it runs in seconds, and it costs cents to judge. It is not a fixed test file — it is a living asset. The user-feedback loop (you’ll wire a feedback_rating field below) is where the next failing cases come from: customers rate answers, you triage the low ratings, and the genuine failures become new golden entries. Rating → triage the failures → grow the golden set.

⚠️ Common misconception: “RAG evaluation and model evaluation are the same thing.” They measure different objects. Model evaluation scores a bare model on public benchmarks (MMLU, summarization sets) — it tells you how a model does in general. RAG evaluation scores your whole system — retrieval plus generation, on your data — and tells you whether Relay answers CloudCart’s tickets correctly. A great MMLU score says nothing about whether your Knowledge Base retrieves the right chunk for “my storefront returns a 500.” Always evaluate the system on your golden set, not the model on someone else’s benchmark.

Choosing the right eval for each component

There is no single magic “is it good?” score. Relay is a router (M3), a Knowledge Base (M5), an agent (M7), and a triage step (M2), and each component fails differently, so each needs a different evaluation. The four tools you will reach for:

Eval type	What it measures	Relay component	When to use	Cost	Main limitation
Bedrock Model Evaluations	A bare model on generic benchmarks or your own prompt set	Router / model choice (M3)	Picking or comparing models before they touch your data	Tokens only, no job surcharge	A benchmark score doesn’t predict quality on your tickets
RAG evaluation (retrieve / retrieve-and-generate)	Retrieval + generation on your data: context relevance (retrieve-only); correctness, faithfulness, completeness, citation precision (retrieve-and-generate)	Knowledge Base + answer (M5)	Whenever answers feel ungrounded or retrieval looks off	Tokens of the evaluator + generations	Needs a managed Knowledge Base; tells you what regressed, not why (that’s M14)
Agent evaluation	Task completion and tool usage — did it call the right tools and no useless ones?	Strands agent (M7)	When the agent acts but you suspect wasted or wrong tool calls	Tokens of the judge over trajectories	Trajectory scoring is harder to calibrate than a single answer
Custom LLM-as-a-judge	Anything you can write a rubric for: triage match, coverage, grounding, fairness	Triage (M2), answer (M5), agent (M7)	When you need a metric no managed metric gives you	Tokens of the judge (cheap on Flex/batch)	An uncalibrated judge scores with its own biases

The decision tree is simpler than the table makes it look — start from what you are evaluating:

flowchart TD
    A[What are you evaluating?] --> B{Picking a model<br/>before it sees your data?}
    B -->|yes| C[Bedrock Model Evaluations<br/>generic benchmarks / your prompts]
    B -->|no| D{Is retrieval + generation<br/>on YOUR data the question?}
    D -->|yes| E[RAG evaluation<br/>correctness, faithfulness, completeness, citation precision]
    D -->|no| F{Is an agent calling<br/>tools to complete a task?}
    F -->|yes| G[Agent evaluation<br/>task completion + tool usage]
    F -->|no| H{Need a custom metric<br/>no managed eval gives you?}
    H -->|yes| I[Custom LLM-as-a-judge<br/>your rubric: triage, coverage, fairness]
    H -->|no| J[A managed eval already fits — use it]

Two things sit on the edge of this picture and stay there. A/B and canary testing (skill 5.1.2) compares two live versions on real traffic: A/B splits requests between variants and compares outcomes; canary testing routes a small slice to the new version first and watches it before a full rollout. It is exam-relevant, but its infrastructure lives in the Module 11 pipeline — here it is theory, not a build. And AgentCore Evaluations (2026) can score agent trajectories directly; it exists, but Relay’s agent eval is a custom judge over the golden set, not a managed trajectory job.

LLM-as-a-judge: power and pitfalls

LLM-as-a-judge is the technique at the center of this module: you give a foundation model an explicit rubric — criteria, a scale, and what each criterion means — and ask it to score another model’s output as structured JSON. It is cheaper and faster than a human panel and far sharper than a string match, because it can recognize that fifty different phrasings of the same correct answer are all correct. The exam treats it as the multi-perspective evaluation lever (skill 5.1.5), alongside RAG evaluation and human feedback.

Relay’s judge rubric evaluates six criteria; four are serialized per ticket. The four persisted scores are: did the triage intent match (triage_ok), how many expected points the answer covered (coverage), whether every claim is supported by the cited sources (grounding), and whether citations were present when required (citations). The other two are the agent-trajectory criteria the judge applies on agent tickets: whether the agent called the right tools and no useless ones (tool usage, skill 5.1.7), and whether the customer’s actual need was met (task completion). The distinction between those last two matters and the exam loves it: an agent can complete the task and call three pointless tools on the way. Task completion alone scores that a 5; tool-usage effectiveness catches the waste.

The non-negotiable rule — a hard invariant in both code and prose — is that the judge is never the candidate. Relay answers with Amazon Nova (fast/smart/vision tiers); the judge is Anthropic Claude Haiku 4.5, a different model family. Using a model to judge its own output imports self-preference bias: it rates “its own” style higher (this is one of the position, verbosity, and self-enhancement — a.k.a. self-preference — biases documented in the Judging LLM-as-a-Judge paper). My opinion, and the design choice: crossing vendors costs nothing and kills the bias argument outright, so do it. A second pitfall is verbosity bias — judges tend to prefer longer answers — which the rubric counters explicitly (“a terse correct answer beats a long vague one; do not reward verbosity”).

⚠️ Common misconception: “An LLM judge replaces human evaluation.” No. An uncalibrated judge scores with its own biases — self-preference, verbosity, position. You earn the right to trust it by calibrating it: hand-score about five cases, then check the judge lands within a point of your scores (aim for ≥ 0.8 agreement) before its scores gate anything. And you keep humans in the loop — the user-feedback ratings (skill 5.1.3) are exactly where new failing cases and recalibration come from. The judge scales human judgment; it does not retire it.

The judge runs deterministically (temperature 0, so it cannot manufacture phantom regressions), and its output is validated through a Pydantic schema with one retry that feeds the validation error back — never a silent try/except that returns a fabricated score. Cost-wise, an eval job tolerates latency, so the judge is meant to run on the Flex tier (−50%). One live finding to record honestly (more in the lab): on the generation date, Claude Haiku 4.5 served only the default service tier, so the judge here bills at standard — the judge ≠ candidate invariant outranks the cost lever, every time.

Fairness via judge (skill 3.4.2) is the same machinery with a different rubric. You run Relay on twin tickets — the same problem, one irrelevant customer attribute changed (name, region, business size, language fluency, emotional tone) — and the judge scores each answer’s quality and tone blind to the attribute. If the two scores diverge by more than one point, answer quality moved with something it should not have. This is the judge-borne half of fairness; the CloudWatch bias metrics and A/B Prompt Management half lives in Module 10’s model card.

From scores to gates: continuous evaluation

A score that blocks nothing protects nothing. The point of the harness is the regression gate: a committed baseline, named runs, and an automatic comparison that fails a deploy when quality drops. Relay commits evals/results/run-baseline.json. Every new run produces run-<name>.json and is compared against it. The gate fails when aggregate grounding falls below 0.8 or when grounding regresses more than 5 points versus the baseline (the frozen evaluation contract, below). That 0.8 is not a new number — it is the same GROUNDING_THRESHOLD constant the Module 9 grounding escalation uses and the Module 14 relay-ops alarm will reuse, surfaced to the gate as config.EVAL_GROUNDING_FLOOR (one named alias of the M9 constant — EVAL_GROUNDING_FLOOR = GROUNDING_THRESHOLD). Defined once, never a divergent literal.

Where the gate lives is the whole point: it is an eval stage in the Module 11 CodePipeline, after the smoke tests and before promotion. The same harness also handles deployment validation (skill 5.1.9) — a fresh deploy runs the golden set against the live system and is allowed to promote only if grounding holds.

flowchart LR
    A[golden_set.json<br/>20 tickets] --> B[run_evals.py<br/>candidate + judge]
    R[Bedrock RAG eval<br/>on relay-kb] -.folds in.-> B
    B --> C[run-name.json<br/>scores + aggregate + cost]
    C --> D{compare to<br/>run-baseline.json}
    D -->|grounding >= 0.8<br/>and drop <= 5pts| E[GATE PASS -> deploy]
    D -->|grounding < 0.8<br/>or drop > 5pts| F[GATE FAIL -> deploy BLOCKED]

The reporting (skill 5.1.8) is deliberately simple: the per-ticket table run_evals.py prints is the report — one row per ticket plus an aggregate line. That is enough to answer the regression question and to brief a stakeholder. Turning those scores into a live dashboard with drift alarms is Module 14’s job; here the table is the deliverable.

The frozen evaluation contract — reproduce field-for-field in the golden set, the run output, and any downstream brief:

evals/golden_set.json = list[{id, ticket: Ticket, expected_intent,
                              expected_points: list, must_cite: bool}]   # 20 entries

uv run python evals/run_evals.py --out evals/results/run-<name>.json
  -> {run_name, config,
      scores: [{id, triage_ok, grounding, coverage, citations}],
      aggregate, cost_cents}

# Regression gate: fail if aggregate.grounding < 0.8 OR regression > 5 pts vs baseline.
# Judge model must never equal the candidate model.

Build it: Relay’s eval harness

The lab adds an evals/ package, a feedback endpoint, and one field to a frozen schema — nothing inherited is rewritten. At the end, one offline command builds the committed baseline without spending a cent, and a degraded-prompt fixture makes the gate fail with a non-zero exit. The full code lives in the Module 13 lab; here are the load-bearing excerpts.

The golden set. Each entry is the frozen contract shape — a real ticket plus what a good answer must contain:

{
  "id": "gold-01-duplicate-charge",
  "ticket": {
    "ticket_id": "gold-01", "channel": "email",
    "customer_message": "I was charged $49 twice for my Pro plan this billing cycle...",
    "created_at": "2026-06-01T09:14:00Z"
  },
  "expected_intent": "billing",
  "expected_points": [
    "two charges with the same amount and order number are a true duplicate",
    "refund the later charge from Billing -> Transactions",
    "a refund reaches the customer in 5-10 business days"
  ],
  "must_cite": true
}

The judge. A rubric scored as Pydantic-validated JSON, with one retry that feeds the validation error back. The judge tier and service tier come from relay/config.py — never a model ID here:

def _call_judge(system, user, schema, *, converse_fn=None):
    converse_fn = converse_fn or llm.converse        # the single Bedrock call site
    prompt = user
    for attempt in (1, 2):
        result = converse_fn(
            [{"role": "user", "content": [{"text": prompt}]}],
            tier=config.JUDGE_TIER,                  # judge != candidate (config enforces)
            system=[{"text": system}],
            inferenceConfig={"maxTokens": 600, "temperature": 0.0},  # deterministic
            service_tier=config.JUDGE_SERVICE_TIER,  # Flex when the model supports it
        )
        try:
            verdict = schema.model_validate_json(_extract_json(result.text))
        except ValidationError as err:
            if attempt == 2:
                raise JudgeError(...)                # never a fabricated score
            prompt = user + f"\nYour reply failed validation:\n{err}\nReturn corrected JSON."
            continue
        return verdict, usage

The judge ≠ candidate invariant lives in config.py — it raises rather than silently scoring with a self-preferring judge:

JUDGE_CANDIDATE_TIERS = frozenset({"fast", "smart", "vision"})  # the answering models

def judge_profile() -> str:
    profile = tier_profile(JUDGE_TIER)
    candidate_profiles = {TIERS[t] for t in JUDGE_CANDIDATE_TIERS if t in TIERS}
    candidate_profiles |= {ALT_PROFILES[t] for t in JUDGE_CANDIDATE_TIERS
                           if t in ALT_PROFILES}    # also block the global. alternates
    if profile in candidate_profiles:
        raise ValueError(
            "Judge profile equals a CANDIDATE tier profile. The judge must never be the "
            "model that produced the answer (self-preference bias)."
        )
    return profile

The gate reads aggregate grounding against the one 0.8 floor and the 5-point regression limit:

def evaluate_gate(result, baseline):
    grounding = float(result["aggregate"]["grounding"])
    passed, reasons = True, []
    if grounding < config.EVAL_GROUNDING_FLOOR:           # alias of GROUNDING_THRESHOLD = 0.8
        passed = False
        reasons.append(f"grounding {grounding:.3f} below floor {config.EVAL_GROUNDING_FLOOR}")
    if baseline is not None:
        drop = float(baseline["aggregate"]["grounding"]) - grounding
        if drop > config.EVAL_REGRESSION_MAX_DROP:        # additive M13 constant: > 5 pts vs baseline
            passed = False
            reasons.append(f"grounding dropped {drop:.3f} vs baseline")
    return GateResult(passed=passed, reasons=reasons)

The feedback loop (skill 5.1.3) adds feedback_rating: int | None = None to TicketRecord by addition — no existing field renamed, retyped, or removed — and a POST /tickets/{id}/feedback handler that wraps the unchanged store, runs no foundation model, and validates a strict 1–5 integer:

def parse_feedback(body: dict) -> int:
    if "feedback_rating" not in body:
        raise common.BadRequest("Body must include an integer 'feedback_rating' (1-5).")
    rating = body["feedback_rating"]
    # bool is an int subclass in Python — reject true/false before the int check.
    if isinstance(rating, bool) or not isinstance(rating, int):
        raise common.BadRequest("'feedback_rating' must be a JSON integer (1-5).")
    if not (FEEDBACK_MIN <= rating <= FEEDBACK_MAX):
        raise common.BadRequest(f"'feedback_rating' must be between 1 and 5 (got {rating}).")
    return rating

Run it. Build the committed baseline offline from a fixture (deterministic, no tokens), then do a real run with the gate:

# Committed baseline — offline, no tokens:
uv run python evals/run_evals.py --fixture data/eval_fixtures/baseline_fixture.json \
  --out evals/results/run-baseline.json

# A real run + the gate (spends tokens):
uv run python evals/run_evals.py --live --gate --out evals/results/run-latest.json

The offline baseline aggregates clean — every fixture answer is cited, and grounding aggregates to 0.963 (a few tickets land at 0.750, the rest at 1.000):

=== Relay eval run: baseline ===
ticket                                   kind        triage  ground  cover   cite
--------------------------------------------------------------------------------
gold-01-duplicate-charge                 nominal     ok      1.000   1.000   yes
gold-07-package-lost                     nominal     ok      0.750   0.750   yes
gold-17-adversarial-injection-exfil      adversarial ok      1.000   1.000   yes
...
--------------------------------------------------------------------------------
AGGREGATE  triage_accuracy=1.000  grounding=0.963  coverage=0.963  citation_rate=1.000
COST       cost_cents=0.3600 ($0.0036)

The committed baseline grounding is 0.963. Now demonstrate the catch. A provided degraded prompt (data/degraded_prompt.md) tells the model to answer from memory and drop citations; its fixture run scores grounding 0.400 — below the 0.8 floor and a 56-point drop:

uv run python evals/run_evals.py --fixture data/eval_fixtures/degraded_fixture.json \
  --out evals/results/run-degraded.json --gate --baseline evals/results/run-baseline.json
# AGGREGATE grounding=0.400 -> GATE FAILED -> deployment BLOCKED -> exit 1

That non-zero exit is what the pipeline reads. Wired as an EvalGate stage after Smoke in the Module 11 CodePipeline (Source → Build → Deploy → Smoke → EvalGate), a grounding regression fails the stage and blocks promotion — automatically, with no human rereading five tickets.

The managed cross-check. setup.py --submit-eval submits a Bedrock RAG evaluation job on relay-kb (retrieve-and-generate), evaluated by Claude Haiku 4.5, over 19 prompts. It ran about 15 minutes to Completed, reporting Correctness 0.556, Faithfulness 0.643, Completeness 0.539, CitationPrecision 0.741. The live home-judge run scored grounding 0.588 over the same golden set — and 0.588 lands right next to the managed Faithfulness 0.643. The two grounding views agree: an independent managed metric corroborating your custom judge, which is exactly what you want. Both fell below the 0.8 floor on this live run, so the live gate correctly blocked — the regression gate working as designed (more in the lab).

Try it yourself

1. Add a conciseness criterion to the rubric and re-baseline. Does a verbose-but-correct answer now score lower? (That is one way to counter the judge’s verbosity bias.)
2. Switch the judge to Nova 2 Lite as an experiment — config.judge_profile() will refuse it as a candidate collision, which is the lesson. Compare two judges’ scores on five tickets: inter-judge calibration.

In production

A 20-ticket golden set is the on-ramp, not the destination. In a real deployment it grows to 200+ entries, fed by the feedback loop and your escalation queue: every low rating and every human handoff is a candidate failing case. You run shadow evals on sampled production traffic, not just the committed set, so drift shows up on inputs you never anticipated. Eval cost becomes its own budget line — backfilling a grown golden set across model versions is exactly the latency-tolerant batch-inference (−50%) work Module 12 set up, and the judge belongs on Flex when the model supports it. The judge rubric should be owned by the product owner, not only the engineer: it encodes what “a good answer” means for the business, and that is a product decision. Recalibrate every time you change the judge model — a new judge has new biases, and an uncalibrated swap silently moves every score. And a judge model upgrade is itself a change that should pass the gate before it ships. In Module 14 you’ll watch these same scores drift in production and alarm on them.

Exam corner

Task 5.1 is Domain 5’s evaluation half (Module 14 carries 5.2, troubleshooting). The exam tests whether you can pick the right evaluation for a component, spot a biased judge, gate a deploy automatically, score an agent on tool usage rather than task completion alone, and prove fairness with controlled pairs rather than spot-reading.

1. A team’s RAG application is returning answers that sound right but are not supported by their documents. They want the most direct first step to measure the problem. Which evaluation should they run?

   • A. A Bedrock Model Evaluation comparing two base models on MMLU.
   • B. A Bedrock RAG evaluation (retrieve-and-generate) on their Knowledge Base, scoring faithfulness and correctness.
   • C. A latency benchmark of the retrieval call.
   • D. A human read of ten sample answers.

2. Your LLM-as-a-judge consistently scores answers from one model family higher than answers from another, even when blind reviewers rate them equal. The judge is from that same favored family. What is happening, and what is the fix?

   • A. Verbosity bias; shorten the rubric.
   • B. Self-preference bias; move the judge to a different model family and calibrate it against human scores.
   • C. Position bias; randomize the answer order.
   • D. Nothing — a consistent judge is a good judge.

3. After a prompt change, you need to automatically prevent a quality-regressed version from reaching production. Which mechanism does that?

   • A. A code review checklist that asks the reviewer to spot regressions.
   • B. A golden set plus a regression gate in CI that fails the build on a grounding drop.
   • C. Post-deployment monitoring that alarms after users complain.
   • D. A larger model for the answer step.

4. An agent returns correct answers but calls three unrelated tools on every request, inflating cost and latency. Your task-completion score is a perfect 5. Which evaluation metric surfaces the problem?

   • A. Task completion.
   • B. Tool-usage effectiveness — did it call the right tools and no useless ones?
   • C. Citation precision.
   • D. Fluency.

5. You must prove your assistant answers with the same quality regardless of the customer’s name, region, or business size. What is the rigorous approach?

   • A. Read a random sample of answers and judge by feel.
   • B. Run a fairness evaluation: a judge scores twin tickets (same problem, one irrelevant attribute changed) and you flag any quality divergence beyond a tolerance.
   • C. Turn on a guardrail.
   • D. Average the customer satisfaction score across segments.

Answers. 1 — B. Ungrounded answers on your data are a system problem, so you score the system: a RAG evaluation on the Knowledge Base, measuring faithfulness and correctness, tells you whether retrieval-plus-generation is grounded. A model eval on MMLU (A) scores a bare model on irrelevant benchmarks; a latency benchmark (C) measures the wrong axis; ten hand-read samples (D) are not reproducible or aggregable. 2 — B. A judge favoring its own model family is the textbook self-preference bias; the fix is to cross model families (judge ≠ candidate) and calibrate against human scores. Verbosity (A) and position (C) are different biases with different signatures, and “a consistent judge is a good judge” (D) confuses consistency with correctness — a consistently biased judge is consistently wrong. 3 — B. Only a golden set plus a CI regression gate blocks a regressed version automatically. A review checklist (A) and monitoring-after-complaints (C) are manual or post-hoc; a bigger model (D) does not gate anything. 4 — B. Task completion alone scores wasted tool calls a perfect 5; tool-usage effectiveness is the metric that penalizes calling tools the ticket did not need. Citation precision (C) and fluency (D) measure other things. 5 — B. A fairness evaluation over controlled twin pairs, scored by a blind judge with a divergence tolerance, is the rigorous, repeatable approach. Spot-reading (A) is not rigorous; a guardrail (C) blocks content, it does not measure fairness; a satisfaction average (D) hides per-segment gaps.

Traps to avoid.

• A public benchmark score does not predict quality on your tickets. MMLU or a summarization leaderboard scores a model in the abstract. Model evaluation ≠ system evaluation on a business golden set. The exam offers the benchmark as a tempting wrong answer.
• The same model as judge and candidate is the bias, not the consistency. “Use one model for both to keep scoring consistent” is exactly the self-preference trap. Cross model families.
• Evals without a baseline and a gate protect nothing. Numbers that block no deploy are decoration. The protective mechanism is the gate, not the score. (And remember: the other half of Domain 5 — troubleshooting, Task 5.2 — is Module 14.)

Key takeaways

• FM-quality metrics are relevance, factual accuracy, consistency, and fluency — not classic ML accuracy. A good answer has no single label to diff against.
• One eval per component, not a single magic score — model eval, RAG evaluation, agent eval, and a custom judge each answer a different question.
• The judge is never the candidate — crossing model families costs nothing and kills self-preference bias; calibrate against human scores before you trust it.
• The golden dataset is a versioned asset, not a test file — it lives, fed by user feedback and escalations, and 20 entries is a fine start.
• No gate means no evaluation — just decorative numbers — the regression gate (grounding < 0.8 or a > 5-point drop) is the part that protects production.
• A managed RAG evaluation should agree with your home judge — Relay’s live judge grounding (0.588) landed next to the managed Faithfulness (0.643); independent corroboration is the goal.
• Evaluation costs tokens — judge on Flex when the model supports it, backfill on batch (−50%); budget it as its own line.

What’s next

Relay now proves its quality before every deploy — but who’s watching it at 3 a.m.? The gate catches a regression you ship; it says nothing about the answer quality, cost, and latency drifting in production between deploys. Module 14 wires up observability: invocation logging, a relay-ops dashboard, four alarms (including a grounding alarm that reuses this module’s 0.8 constant), X-Ray tracing, and a troubleshooting runbook — the other half of Domain 5.

Want the full AIP-C01 question bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Module 13 lab · ← Module 12 · Course index · Module 14 →

References

• Evaluate the performance of Amazon Bedrock resources — Amazon Bedrock — model evaluation and RAG evaluation jobs, including the LLM-as-a-judge evaluator and the metric definitions.
• Evaluate a knowledge base (RAG evaluation) — Amazon Bedrock — retrieve evaluations (context relevance, retrieve-only) and retrieve-and-generate evaluations (correctness, faithfulness, completeness, citation precision).
• Amazon Bedrock pricing — evaluation jobs (tokens only, no surcharge), service tiers (Flex −50%), and per-token figures (dated; re-verify on generation day, as of June 2026).
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., arXiv 2306.05685) — the position, verbosity, and self-enhancement (a.k.a. self-preference) biases that motivate judge ≠ candidate and calibration.
• aws/fmeval — AWS’s open-source evaluation library; a code-level alternative to managed jobs (release cadence is slow — managed Bedrock Evaluations plus a custom judge is this course’s path).
• AWS Certified Generative AI Developer - Professional (AIP-C01) exam guide — Task 5.1 (evaluation) and skill 3.4.2 (fairness), the skills this module covers.