Evals and judges

A recipe's evals are its judges. They are declared source in judges/, validated with the rest of the package before a session starts, and they travel with the recipe, so any runtime can grade live conversations against the same rubric you wrote locally.

Everything else you write to test the agent stays yours. evals/ is not a Recipe directory and nothing validates it, so lay it out however your runners prefer. What it gains by living here is that the thing measuring your agent is versioned with the agent it measures, and travels with it.

my-recipe/
  evals/                     # yours. Not a Recipe directory, never validated.
    brief-shape.eval.ts      # Evalite: fast, in-process, on every commit
    tasks/
      acme-q2/               # Harbor: a container, a verifier, a reward
        task.toml
        instruction.md
        environment/Dockerfile
        tests/test.sh
  judges/                    # declared source. Checked, and deployed.
    sourced.yaml             # direct children only: judges/x/y.yaml is not a judge
    trajectory.yaml          # graded over the whole run, not the output

Lightweight evals

Reach for Evalite when the thing you are checking is one step with a checkable output: a skill's formatting rule, an extraction, a classification. It runs in-process in seconds, so it belongs on every commit.

import { evalite } from "evalite";
import { Levenshtein } from "autoevals";

evalite("Brief opens with a recommendation", {
  // Cases in, expected out.
  data: [{ input: "acme-q2", expected: "Recommend: expand" }],

  // The thing under test. Call your recipe, or one piece of it.
  task: async (input) => runBrief(input),

  // Any scorer: string distance, a regex check, an LLM judge.
  scorers: [Levenshtein],
});

npx evalite watch evals/   # re-runs on save, while you work
npx evalite run evals/     # once, for CI

Agent evals

Reach for Harbor when success depends on the whole environment and several decisions in a row, which is the case a unit test cannot cover honestly. A task is a directory: an instruction, a container, and a verifier that writes a reward.

schema_version = "1.4"

[task]
name = "acme/research-brief"
version = "1.0.0"
description = "Produce a sourced brief from a fixed CRM snapshot"

[verifier]
timeout_sec = 120.0

[agent]
timeout_sec = 600.0        # long-horizon work needs the headroom

[environment]
os = "linux"
cpus = 1
memory_mb = 2048

#!/bin/bash
# The verifier runs in its own container and writes a reward file.
# Harbor reads reward.json first and falls back to reward.txt.

uvx pytest /tests/test_brief.py

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

harbor run evals/tasks/acme-q2   # builds the container, runs the verifier

reward.json lets a verifier report several metrics rather than one bit, which is what you want when a task can partly succeed.

Judges

A judge is a rubric over one meaning-dependent decision, graded by a model. Keep the scope narrow: a judge that asks whether the whole answer was good produces a number nobody can act on.

judge: sourced      # unique in the recipe
description: Does every material claim carry a dated source?

instructions: |
  Read the brief. For each factual claim about the account,
  check that a source and a date are given. Pass only when
  every claim is sourced. Fail a brief that reads well and
  cites nothing.

llm:
  provider: openai
  model: gpt-5
  request:
    temperature: 0     # the default, and what a repeatable score needs

• judge, instructions and llm.model are required and non-empty. Everything else has a default.
• Names are unique across the package, and only direct children of judges/ are judge sources. judges/calibration/sourced.yaml is a fixture, not a second judge.
• Unknown fields are errors at every level, so a mistyped key fails the check instead of being quietly ignored.

Nothing in the package runs a judge. Your runtime reads the YAML and calls the model, which is the whole point of the split: the rubric is reviewed and versioned with the behavior it grades, and the same definition serves twice. Locally it is a reference while you develop. In your runtime it is an online eval over live conversations, or a hook on a run that finishes.

Gating a judge to what it can score

Without on:, a judge applies to every conversation selected for judging. Add it and the judge is an OR-list of matchers over message, tool and feedback events, where the fields inside a single match all have to hold. A regex literal uses Rust syntax and takes the i, m, s and u flags.

on:
  - event: message
    match:
      role: user
      text: /refund|invoice/i     # regex literal
  - event: tool
    match:
      name: shell
      args.command: /pytest/i     # dotted paths reach into the payload
  - event: feedback
    match:
      sentiment: negative

Trajectory judges

An output judge reads the answer. A trajectory judge reads how the agent got there, which is the only way to catch a run that reached a plausible answer by a route you would reject: the CRM never opened, a tool retried nine times, a subagent's finding silently dropped.

The dimensions worth separating are planning, execution order, tool selection and arguments, adaptation after a bad result, efficiency, and whether constraints held for the whole run. Grade one rubric item per judgement call rather than asking for one score across all six, because a single number tells you nothing about which one failed.

judge: sources-actually-opened
description: Did the agent read the sources it cites?

# One rubric item, one verdict. Write one file per item.
instructions: |
  You are given the task and the agent's full step history.
  Every source cited in the final brief must appear as a
  successful read or fetch earlier in the trajectory.
  Fail when a citation has no matching tool call, even if
  the claim happens to be true.

llm:
  provider: openai
  model: gpt-5
  request:
    temperature: 0

Formatting a trajectory

A judge can only be as good as the history you hand it, and an ad-hoc log shape means every judge is coupled to your runtime. Harbor's Agent Trajectory Interchange Format is the standard shape, and it is worth emitting even if Harbor is not your runner, because the same file then feeds debugging, judging, and fine-tuning.

{
  "schema_version": "ATIF-v1.7",
  "agent": { "name": "acme-research", "version": "1.2.0" },
  "steps": [
    { "step_id": 1, "source": "user", "message": "Research NordStream" },
    {
      "step_id": 2,
      "source": "agent",
      "reasoning_content": "Start with the CRM record.",
      "tool_calls": [
        {
          "tool_call_id": "call_1",
          "function_name": "read",
          "arguments": { "path": "crm-4471.json" }
        }
      ],
      "observation": {
        "results": [
          { "source_call_id": "call_1", "content": "{ \"account\": … }" }
        ]
      }
    }
  ],
  "final_metrics": { "total_cost_usd": 0.42 }
}

What makes it judgeable is that every observation.results[].source_call_id points back at a tool_call_id, so a judge can check that a cited source was actually read rather than inferring it from prose. reasoning_content is what lets a judge separate a bad plan from bad execution. Keep every step, including the failed calls: a trajectory with the mistakes edited out cannot be graded on adaptation. subagent_trajectories holds a delegated run as a complete nested trajectory, which is how you judge whether a child was given the right task.

The GenAI spans Recipes emits are the same information in a different shape, and either can be projected into the other. Pick one and be consistent, because a judge calibrated on one shape does not transfer.

Calibration

A judge you have not calibrated is an opinion. Label a set of real conversations by hand, split them into train, dev and test, tune the rubric against train and dev, then measure agreement once on the held-out test split. Adjusting labels so the judge scores better defeats the exercise.

Calibration fixtures belong beside the judge, so a rubric and the evidence that it agrees with a human are one reviewable change. The definition you calibrated locally is the one your runtime grades with, which is what makes a local pass and a production judgement mean the same thing.