Evaluators¶

Evaluators: score traces with deterministic checks or LLM judges (docs/design.md §5).

An evaluator is anything with a name and evaluate(trace, case) -> EvalResult (Evaluator). The quickest way to write one is the decorator:

import agentlens

@agentlens.evaluator
def mentions_order(trace, case) -> bool:
    return "order" in str(trace.output)

Suites refer to evaluators by name through the registry, using the YAML spellings of EvaluatorSpec:

from agentlens.evals import evaluate_case, get_evaluator

check = get_evaluator("mentions_order")
results = evaluate_case(trace, case, ["mentions_order"])

Built-in evaluators (agentlens.evals.builtin: exact_match, contains, regex, json_schema, no_error, tool_sequence, max_steps, cost_budget, latency_budget, and the LLM judges llm_rubric / llm_pass_fail) register themselves in default_registry when this package is imported:

results = evaluate_case(trace, case, ["no_error", {"contains": "refund"},
                                      {"regex": {"pattern": r"order \d+"}},
                                      {"tool_sequence": {"mode": "ordered"}},
                                      {"max_steps": 8}])

LLM judges (agentlens.evals.judge) call their judge model at temperature 0, parse its JSON verdict robustly and cache it by sha256(judge model, prompt, output):

judge = get_evaluator({"llm_rubric": "Polite, confirms the refund amount"},
                      model="openai:gpt-4o-mini")

PairwiseJudge compares two traces of the same case (baseline vs candidate), judging both presentation orders to cancel position bias:

result = PairwiseJudge("Confirms the refund amount", model="openai:gpt-4o-mini").compare(
    baseline_trace, candidate_trace, case)
result.winner, result.to_eval_result(candidate="b").passed

An evaluator that raises never crashes a run: run_evaluator and @evaluator turn the exception into a failed EvalResult. Configuration errors (unknown name, bad parameters) are raised eagerly.

DEFAULT_THRESHOLD `module-attribute` ¶

DEFAULT_THRESHOLD = 0.5

Pass threshold for evaluators that return a bare score: passed = score >= threshold.

UNSET `module-attribute` ¶

UNSET: Final = _Unset.UNSET

Default of ExactMatch's value: "not given, use case.expected" (distinct from None, which expects a null output).

OutputPath `module-attribute` ¶

OutputPath: TypeAlias = str | Sequence[str | int]

Where to look inside a structured output: "answer", "items.0.text" or ["weird.key", 0] (the list form for keys that contain dots).

EvaluatorFactory `module-attribute` ¶

EvaluatorFactory: TypeAlias = Callable[..., Evaluator]

factory(**params) -> Evaluator; an evaluator class is a valid factory.

SpecLike `module-attribute` ¶

SpecLike: TypeAlias = EvaluatorSpec | str | Mapping[str, Any]

An EvaluatorSpec or one of its YAML spellings ("no_error", {"max_steps": 8}).

default_registry `module-attribute` ¶

default_registry = EvaluatorRegistry()

The process-wide registry: built-ins and @evaluator functions land here.

AsyncEvaluator ¶

Bases: Evaluator, Protocol

An evaluator with a native async path (e.g. an LLM judge doing network I/O).

arun_evaluator awaits aevaluate when an evaluator has it and otherwise runs the sync evaluate in a worker thread.

aevaluate ¶

aevaluate(trace: Trace, case: Case) -> Awaitable[EvalResult]

Async version of evaluate.

Evaluator ¶

Bases: Protocol

Anything that scores one trace of one case (docs/design.md §5).

name is what ends up in EvalResult.evaluator; it is declared as a read-only property so both a plain attribute and a @property satisfy the protocol. evaluate should return an EvalResult; exceptions are turned into failed results by run_evaluator.

name `property` ¶

name: str

Evaluator name, as used in suite files and results.

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Score trace (the agent's run on case).

Contains ¶

Contains(text: str | list[str] | None = None, *, mode: Literal['all', 'any'] = 'all', case_sensitive: bool = True, normalise_whitespace: bool = False, path: OutputPath | None = None)

contains: the output text contains one or more substrings.

{contains: "refund"} or {contains: {text: [refund, "order 1234"], mode: any}}. Without text the substrings come from case.expected (a string or a list of strings). The output is searched as text (non-strings as canonical JSON).

The score is the fraction of substrings found. mode="all" (default) passes only when every substring is found; mode="any" passes when at least one is. A None output contains nothing.

Parameters:

Name	Type	Description	Default
`text`	`str \| list[str] \| None`	Substring or list of substrings; defaults to `case.expected`.	`None`
`mode`	`Literal['all', 'any']`	`"all"` or `"any"`.	`'all'`
`case_sensitive`	`bool`	`False` compares casefolded text.	`True`
`normalise_whitespace`	`bool`	Treat any run of whitespace (in both) as one space.	`False`
`path`	`OutputPath \| None`	Search only this part of the output.	`None`

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Look for each substring in the output text.

CostBudget ¶

CostBudget(max_usd: float, *, on_unknown: UnknownCostPolicy = 'fail')

cost_budget: the run cost at most max_usd ({cost_budget: 0.05}).

The cost is Trace.total_cost, the sum of usage.cost_usd over the spans (filled from the pricing table; MockLLM reports 0). Score 1 within budget, else max_usd / cost.

Unknown cost. When no span has a cost (an unpriced model), or some spans with usage lack one, the budget cannot be verified. If the known part already exceeds the budget the check fails anyway; otherwise on_unknown decides: "fail" (default — a budget you cannot check is not a budget you met, and a model missing from the pricing table should be noticed) or "pass" (score 1, with the reason saying the cost is unknown).

Parameters:

Name	Type	Description	Default
`max_usd`	`float`	Budget in US dollars (`>= 0`).	required
`on_unknown`	`UnknownCostPolicy`	`"fail"` or `"pass"` when the cost is unknown or partial.	`'fail'`

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Compare Trace.total_cost with the budget.

ExactMatch ¶

ExactMatch(value: JsonValue | _Unset = UNSET, *, case_sensitive: bool = True, strip: bool = True, normalise_whitespace: bool = False, path: OutputPath | None = None)

exact_match: the output equals an expected value.

The expected value is the value parameter when given, else case.expected ({exact_match: "Paris"} overrides the case). With neither, evaluate raises EvaluatorConfigError.

Text comparison when the expected value is a string: a non-string output is compared as canonical JSON text. Both sides are NFC-normalised, then strip (default on), normalise_whitespace and case_sensitive apply.
Structural comparison when the expected value is anything else (dict, list, number, bool, null): a string output is parsed as JSON first, so '{"a": 1}' matches {a: 1}. Key order does not matter, 1 == 1.0, but true != 1; the text options apply to every string inside.

Score 1 on a match, else 0.

Parameters:

Name	Type	Description	Default
`value`	`JsonValue \| _Unset`	Expected value; defaults to `case.expected`. Pass `None` explicitly to expect a `null` output.	`UNSET`
`case_sensitive`	`bool`	Compare strings case-sensitively (`False` uses casefold).	`True`
`strip`	`bool`	Ignore leading/trailing whitespace.	`True`
`normalise_whitespace`	`bool`	Treat any run of whitespace as one space.	`False`
`path`	`OutputPath \| None`	Compare only this part of the output (see `output_value`).	`None`

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Compare the output with the expected value (see the class docstring).

JsonSchema ¶

JsonSchema(schema: dict[str, JsonValue] | None = None, *, parse_json: bool = True, max_errors: int = 5, path: OutputPath | None = None)

json_schema: the output is valid against a JSON Schema.

The schema is the schema parameter, else case.expected["schema"] ({json_schema: {schema: {type: object, required: [id]}}}). The draft is taken from $schema (default 2020-12); the schema itself is checked when the evaluator is built (invalid → EvaluatorConfigError).

A string output is parsed as JSON first (one surrounding Markdown code fence is tolerated); unparsable text fails with the parse error. Set parse_json=False to validate a string output as a JSON string instead. A None output is validated as null. Score 1 when valid, else 0; the reason lists the first max_errors errors (sorted by location) and details["errors"] holds them as {path, message, validator}.

Parameters:

Name	Type	Description	Default
`schema`	`dict[str, JsonValue] \| None`	The JSON Schema; defaults to `case.expected["schema"]`.	`None`
`parse_json`	`bool`	Parse a string output as JSON before validating.	`True`
`max_errors`	`int`	How many validation errors to report (`>= 1`).	`5`
`path`	`OutputPath \| None`	Validate only this part of the output.	`None`

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Validate the (parsed) output against the schema.

LatencyBudget ¶

LatencyBudget(max_ms: float)

latency_budget: the run took at most max_ms ms ({latency_budget: 5000}).

The latency is Trace.latency_ms (wall clock from start to end of the run). A run that has not ended (latency_ms is None) fails with score 0. Score 1 within budget, else max_ms / latency.

Parameters:

Name	Type	Description	Default
`max_ms`	`float`	Budget in milliseconds (`>= 0`).	required

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Compare Trace.latency_ms with the budget.

LLMPassFail ¶

LLMPassFail(criteria: str, *, include_expected: bool = True, template: str | None = None, model: str | None = None, client: LLMClient | None = None, cache: CacheSpec = None, retries: int = 1, max_tokens: int | None = None)

Bases: _CriteriaJudge[PassFailVerdict]

llm_pass_fail: an LLM judge decides whether the output meets all criteria.

The reply's verdict ("pass"/"fail"; yes/no, true/false and booleans are tolerated) gives score 1 or 0; reasoning becomes the result's reason. The prompt (prompts/pass_fail.txt) tells the judge to fail when unsure.

Parameters:

Name	Type	Description	Default
`criteria`	`str`	The conditions the output must meet.	required
`include_expected`	`bool`	Show `case.expected` to the judge as a reference.	`True`
`template`	`str \| None`	Custom prompt text (same placeholders as `LLMRubric`; it must ask for `verdict`/`reasoning` JSON).	`None`

model, client, cache, retries and max_tokens are as in LLMJudge.

parse_reply ¶

parse_reply(data: dict[str, JsonValue]) -> PassFailVerdict

Read verdict and reasoning.

make_result ¶

make_result(verdict: PassFailVerdict, run: JudgeRun[PassFailVerdict]) -> EvalResult

Score 1 on pass, 0 on fail.

LLMRubric ¶

LLMRubric(criteria: str, *, min_score: float = 4, include_expected: bool = True, template: str | None = None, model: str | None = None, client: LLMClient | None = None, cache: CacheSpec = None, retries: int = 1, max_tokens: int | None = None)

Bases: _CriteriaJudge[RubricVerdict]

llm_rubric: an LLM judge scores the output from 1 to 5 against criteria.

The 1-5 score is normalised to [0, 1] as (score - 1) / 4 (so 4 → 0.75) and the result passes when score >= min_score. The prompt anchors each point of the scale (prompts/rubric.txt). A fractional score (3.5) is kept; a score outside 1-5 is a parse error (retried, then a failed result).

Parameters:

Name	Type	Description	Default
`criteria`	`str`	What a good output does (`{llm_rubric: "Polite, confirms the amount"}`).	required
`min_score`	`float`	Lowest passing 1-5 score (default 4: "meets the criteria with minor issues").	`4`
`include_expected`	`bool`	Show `case.expected` to the judge as a reference.	`True`
`template`	`str \| None`	Custom prompt text with `$criteria`, `$input`, `$output`, `$reference` placeholders (it must still ask for `score`/`reasoning` JSON).	`None`

model, client, cache, retries and max_tokens are as in LLMJudge.

parse_reply ¶

parse_reply(data: dict[str, JsonValue]) -> RubricVerdict

Read score (1-5, numeric strings accepted) and reasoning.

make_result ¶

make_result(verdict: RubricVerdict, run: JudgeRun[RubricVerdict]) -> EvalResult

Normalise the score and compare it with min_score.

MaxSteps ¶

MaxSteps(value: int)

max_steps: the agent took at most value steps ({max_steps: 8}).

Steps are Trace.n_steps: agent_step spans when the agent records them, otherwise the number of LLM calls. Score 1 within the limit, else value / steps (0 for value=0); passes only within the limit.

Parameters:

Name	Type	Description	Default
`value`	`int`	Maximum number of steps (an integer `>= 0`).	required

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Compare Trace.n_steps with the limit.

NoError ¶

NoError(include_spans: bool = True)

no_error: the agent finished without raising, and (by default) no span failed.

trace.error is set when the agent itself raised. With include_spans (default True) a span that recorded an error — a tool or LLM call that raised, even if the agent caught the exception and carried on — fails the check too: in a regression suite a swallowed tool error is usually a bug worth seeing. Use {no_error: {include_spans: false}} to check only the run itself. (LLM adapter retries happen inside one span, so a retried and recovered HTTP error is not a span error.)

Score 1 without errors, else 0; details holds trace_error, span_error_count and the first failed spans (id, name, kind, error), in start-time order.

Parameters:

Name	Type	Description	Default
`include_spans`	`bool`	Also fail on spans that recorded an error.	`True`

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Look for trace.error and, if enabled, span errors.

OutputPathError ¶

Bases: LookupError

The requested path does not exist in the output (a failed check, not a config error).

Regex ¶

Regex(pattern: str, *, mode: Literal['search', 'match', 'fullmatch'] = 'search', ignore_case: bool = False, multiline: bool = False, dotall: bool = False, path: OutputPath | None = None)

regex: the output text matches a regular expression (Python re syntax).

{regex: "order \d+"} or {regex: {pattern: "^Refund", ignore_case: true}}. The pattern is compiled when the evaluator is built, so an invalid one is an EvaluatorConfigError at suite load. A None output never matches.

Parameters:

Name	Type	Description	Default
`pattern`	`str`	The regular expression.	required
`mode`	`Literal['search', 'match', 'fullmatch']`	`"search"` (anywhere, default), `"match"` (at the start) or `"fullmatch"` (the whole text).	`'search'`
`ignore_case`	`bool`	`re.IGNORECASE`.	`False`
`multiline`	`bool`	`re.MULTILINE` (`^`/`$` match at every line).	`False`
`dotall`	`bool`	`re.DOTALL` (`.` matches newlines).	`False`
`path`	`OutputPath \| None`	Match only this part of the output.	`None`

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Apply the pattern to the output text.

ToolSequence ¶

ToolSequence(tools: Sequence[str] | None = None, *, mode: ToolSequenceMode = 'ordered', allow_extra: bool = True, forbidden: Sequence[str] | None = None)

tool_sequence: the agent called the expected tools (FR-E1).

The actual calls are Trace.tool_calls (names of tool_call spans, in start order). The expected tools are the tools parameter, else case.expected["tools"] (expected: {tools: [lookup_order, issue_refund]} with {tool_sequence: {mode: ordered}}); with neither, evaluate raises EvaluatorConfigError.

Modes (repeats always count — [search, search] needs two search calls):

strict — the calls are exactly the expected list. Score = LCS / max(len(expected), len(actual)), so extra, missing and reordered calls all lower it, and it is 1 only on an exact match.
ordered (default) — the expected list is a subsequence of the calls: in that order, other calls allowed in between. Score = LCS / len(expected).
subset — every expected call happened, in any order (multiset inclusion). Score = matched / len(expected), where each call can match one expected entry.

LCS is the longest common subsequence of the two lists. An empty expected list passes ordered/subset trivially and, in strict, means "no tool calls". Optional constraints, which fail the check when violated:

allow_extra=False (ordered/subset) — no call to a tool outside the expected list (retries of expected tools are fine); each such call is added to the score's denominator. strict already forbids extras.
forbidden — tools that must never be called; any call scores 0.

details hold mode, source, expected, actual, missing (expected entries not matched), unexpected (calls to tools not in the expected list) and forbidden_called.

Parameters:

Name	Type	Description	Default
`tools`	`Sequence[str] \| None`	Expected tool names; defaults to `case.expected["tools"]`.	`None`
`mode`	`ToolSequenceMode`	`"strict"`, `"ordered"` or `"subset"`.	`'ordered'`
`allow_extra`	`bool`	Allow calls to tools that are not expected.	`True`
`forbidden`	`Sequence[str] \| None`	Tools that must not be called at all.	`None`

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Compare Trace.tool_calls with the expected tools (see the class docstring).

FunctionEvaluator ¶

FunctionEvaluator(fn: Callable[..., Any], *, name: str | None = None, threshold: float = DEFAULT_THRESHOLD, params: Mapping[str, JsonValue] | None = None)

An Evaluator backed by a function fn(trace, case, **params).

Created by evaluator. The object stays callable like the original function (handy in unit tests); evaluate calls it with the bound params and normalises the return value with to_eval_result. Any exception — from the function or from an unusable return value — becomes a failed EvalResult (error_result), so one broken evaluator never crashes a suite.

threshold is the pass threshold for bare float scores. It can also be set per spec ({my_eval: {threshold: 0.8}}); the function receives it only if it declares a threshold parameter itself.

Async functions (async def) are supported: aevaluate awaits them and evaluate runs them to completion (in a worker thread when called from inside a running event loop).

bind ¶

bind(**params: JsonValue) -> FunctionEvaluator

A copy with params (merged over the current ones) bound for evaluate.

This is the registry factory: {name: {k: v}} in a suite calls bind(k=v). A threshold param sets the pass threshold.

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Run the function and normalise its result; never raises.

aevaluate `async` ¶

aevaluate(trace: Trace, case: Case) -> EvalResult

Async evaluate: awaits an async function, runs a sync one in a thread.

DuplicateEvaluatorError ¶

Bases: EvaluatorError, ValueError

An evaluator with this name is already registered (pass replace=True to override).

EvaluatorConfigError ¶

Bases: EvaluatorError, ValueError

An evaluator spec is malformed or its parameters do not fit the factory.

EvaluatorError ¶

Bases: Exception

Base class for every evaluator configuration error.

UnknownEvaluatorError ¶

UnknownEvaluatorError(name: str, available: list[str], suggestions: list[str])

Bases: EvaluatorError, LookupError

No evaluator is registered under the requested name.

DiskJudgeCache ¶

DiskJudgeCache(directory: str | PathLike[str] = DEFAULT_JUDGE_CACHE_DIR)

One JSON file per key: <directory>/<key[:2]>/<key>.json.

Files are written atomically (temp file + os.replace), so concurrent runs never read half a file; an unreadable or corrupt file counts as a miss. The directory is created on the first write. Plain files make the cache easy to inspect, delete selectively, and save/restore between CI jobs.

path_for ¶

path_for(key: str) -> Path

The file that holds key.

get ¶

get(key: str) -> CacheEntry | None

The entry for key, or None if missing or unreadable.

set ¶

set(key: str, value: CacheEntry) -> None

Write value to the key's file atomically.

JudgeCache ¶

Bases: Protocol

Where judge replies are kept between calls. Implementations must be thread-safe.

get ¶

get(key: str) -> CacheEntry | None

The entry stored under key, or None on a miss.

set ¶

set(key: str, value: CacheEntry) -> None

Store value (a JSON object) under key, replacing any previous entry.

JudgeParseError ¶

Bases: ValueError

The judge's reply is not the JSON object it was asked for.

LLMJudge ¶

LLMJudge(*, model: str | None = None, client: LLMClient | None = None, cache: CacheSpec = None, retries: int = 1, max_tokens: int | None = None)

Bases: BaseJudge[VerdictT]

Base class of LLM-as-judge evaluators (docs/design.md §5, FR-E2/FR-E4).

A BaseJudge that is also an Evaluator: subclasses add build_prompt (one prompt per trace) and make_result. Parameters are those of BaseJudge.

build_prompt `abstractmethod` ¶

build_prompt(trace: Trace, case: Case) -> JudgePrompt

Render the prompt for trace (the agent's run on case).

make_result `abstractmethod` ¶

make_result(verdict: VerdictT, run: JudgeRun[VerdictT]) -> EvalResult

Turn a parsed verdict into the EvalResult.

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Judge trace synchronously (see the module docstring for the flow).

aevaluate `async` ¶

aevaluate(trace: Trace, case: Case) -> EvalResult

Judge trace with the client's async API.

MemoryJudgeCache ¶

MemoryJudgeCache()

An in-process cache (a dict guarded by a lock). Lost when the process exits.

get ¶

get(key: str) -> CacheEntry | None

The entry for key (a copy), or None.

set ¶

set(key: str, value: CacheEntry) -> None

Store a copy of value under key.

clear ¶

clear() -> None

Forget every entry.

PairwiseJudge ¶

PairwiseJudge(criteria: str, *, include_expected: bool = True, template: str | None = None, model: str | None = None, client: LLMClient | None = None, cache: CacheSpec = None, retries: int = 1, max_tokens: int | None = None)

Bases: BaseJudge[PairwiseVerdict]

An LLM judge that compares two traces of the same case on criteria (FR-E2).

Not an evaluator (it needs two traces): call compare / acompare and, where an EvalResult is needed, PairwiseResult.to_eval_result. Every comparison makes two judge calls (orders "ab" and "ba") — see the module docstring for how they are combined.

Parameters:

Name	Type	Description	Default
`criteria`	`str`	What makes one output better than the other.	required
`include_expected`	`bool`	Show `case.expected` to the judge as a reference.	`True`
`template`	`str \| None`	Custom prompt with `$criteria`, `$input`, `$output_1`, `$output_2` and `$reference` (it must ask for `winner`/`reasoning` JSON).	`None`

model, client, cache, retries and max_tokens are as in BaseJudge.

Raises:

Type	Description
`EvaluatorConfigError`	for invalid parameters.

build_prompt ¶

build_prompt(trace_a: Trace, trace_b: Trace, case: Case | None, order: Order) -> JudgePrompt

Render the prompt for one presentation order.

The input is trace_a.input (else trace_b.input, else case.input). The cached output is the ordered pair [output 1, output 2]: the presentation order is part of the cache key.

parse_reply ¶

parse_reply(data: dict[str, JsonValue]) -> PairwiseVerdict

Read winner ("1"/"2"/"tie") and reasoning.

compare ¶

compare(trace_a: Trace, trace_b: Trace, case: Case | None = None) -> PairwiseResult

Compare trace_a (e.g. the baseline) with trace_b (the candidate).

Asks the judge in both orders (or reads the cache) and averages. A reply that cannot be parsed gives a result with error set; LLM errors propagate.

acompare `async` ¶

acompare(trace_a: Trace, trace_b: Trace, case: Case | None = None) -> PairwiseResult

Async compare: the two orders are judged concurrently.

PairwiseResult `dataclass` ¶

PairwiseResult(winner: Winner, preference: float, consistent: bool, confidence: float, orders: tuple[OrderVerdict, OrderVerdict], criteria: str, error: str | None = None)

The outcome of comparing trace A with trace B over both presentation orders.

Attributes:

Name	Type	Description
`winner`	`Winner`	`"a"`, `"b"` or `"tie"` (from `preference`).
`preference`	`float`	Mean preference for A over the two orders: 1 = A better in both, 0.5 = tie (or opposite answers), 0 = B better in both.
`consistent`	`bool`	Both orders gave the same (mapped) winner.
`confidence`	`float`	Agreement between the orders, `1 - \|pref_ab - pref_ba\|`: 1 when consistent, 0.5 when one order says tie, 0 when they are opposite.
`orders`	`tuple[OrderVerdict, OrderVerdict]`	The two `OrderVerdict` (`"ab"` then `"ba"`).
`criteria`	`str`	What the judge compared on.
`error`	`str \| None`	Set when a reply could not be parsed; the result is then a tie with confidence 0 and `to_eval_result` fails.

ok `property` ¶

ok: bool

Both orders gave a parseable verdict.

reasons `property` ¶

reasons: list[str]

The judge's explanation for each order that gave one ("ab: ...", "ba: ...").

reason `property` ¶

reason: str

One-line summary, e.g. "B is better (preference for A 0.00, consistent)".

combine `classmethod` ¶

combine(ab: OrderVerdict, ba: OrderVerdict, criteria: str) -> PairwiseResult

Average the two orders' preferences (position-bias mitigation).

details ¶

details() -> dict[str, JsonValue]

JSON-ready details: winner, preference, consistency and both orders.

to_eval_result ¶

to_eval_result(candidate: Literal['a', 'b'] = 'b', *, name: str = PAIRWISE_NAME) -> EvalResult

The comparison as an EvalResult for the candidate trace.

Score = the candidate's preference (1 - preference for B, preference for A); passed when the candidate is not worse (it wins or ties). An unparseable comparison fails with score 0 and details["error"].

Raises:

Type	Description
`ValueError`	if `candidate` is not `"a"` or `"b"`.

EvaluatorRegistry ¶

EvaluatorRegistry()

A name -> factory mapping. Most code uses the module-level default_registry.

register ¶

register(name: str, factory: EvaluatorFactory, *, primary: str | None = None, replace: bool = False, signature: Signature | None = None, origin: str | None = None) -> RegisteredEvaluator

Register factory under name.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name used in suite files and results.	required
`factory`	`EvaluatorFactory`	`factory(**params) -> Evaluator` (an evaluator class works).	required
`primary`	`str \| None`	Parameter receiving a scalar shorthand (`{name: x}`); defaults to the first keyword-passable parameter of `factory`.	`None`
`replace`	`bool`	Overwrite an existing entry instead of raising.	`False`
`signature`	`Signature \| None`	Override the introspected signature used to check params.	`None`
`origin`	`str \| None`	Override the definition site used to detect re-registration.	`None`

Raises:

Type	Description
`DuplicateEvaluatorError`	if `name` is taken by a factory from another origin and `replace` is false.
`ValueError`	for an empty name or an unknown `primary` parameter.

unregister ¶

unregister(name: str) -> None

Remove name. Raises UnknownEvaluatorError if it is not registered.

entry ¶

entry(name: str) -> RegisteredEvaluator

The entry for name, or UnknownEvaluatorError with suggestions.

get ¶

get(spec: SpecLike, /, **params: JsonValue) -> Evaluator

Build an evaluator from a spec ("no_error", {"max_steps": 8}, ...).

Keyword params are merged over the spec's own parameters, so get("contains", text="refund") works without writing a spec.

Raises:

Type	Description
`EvaluatorConfigError`	for a malformed spec or parameters.
`UnknownEvaluatorError`	if the name is not registered.

names ¶

names() -> list[str]

Registered names, sorted.

RegisteredEvaluator `dataclass` ¶

RegisteredEvaluator(name: str, factory: EvaluatorFactory, signature: Signature | None, primary: str | None, origin: str | None)

One registry entry.

signature `instance-attribute` ¶

signature: Signature | None

Signature used to check parameters; None when it cannot be introspected.

primary `instance-attribute` ¶

primary: str | None

Parameter that receives a scalar shorthand value ({name: x}).

origin `instance-attribute` ¶

origin: str | None

Where the factory was defined (module.qualname); re-registering from the same origin (a re-run notebook cell, a reloaded module) replaces silently.

build ¶

build(params: Mapping[str, JsonValue] | None = None) -> Evaluator

Call the factory with params (after applying the scalar shorthand rule).

Raises:

Type	Description
`EvaluatorConfigError`	if the parameters do not fit the factory, the factory rejects them (`TypeError`/`ValueError`) or it returns something that is not an `Evaluator`.

error_result ¶

error_result(name: str, exc: BaseException) -> EvalResult

The failed EvalResult that stands in for an evaluator that raised.

Score 0, passed=False, reason = "evaluator error: <Type>: <message>" and details["error"] = {"type": ..., "message": ...} so reports can tell an evaluator crash from a genuine failure.

to_eval_result ¶

to_eval_result(name: str, value: Any, *, threshold: float = DEFAULT_THRESHOLD, reason: str | None = None) -> EvalResult

Normalise what a custom evaluator returned into an EvalResult.

Accepted values:

an EvalResult — returned unchanged;
bool — score 1.0/0.0, passed = the value;
int/float in [0, 1] — the score, passed = score >= threshold;
(value, reason) — any of the above plus a reason string (or None);
a mapping with score and/or passed (plus optional reason, details) — a missing passed is derived from score and threshold, a missing score from passed.

Raises:

Type	Description
`TypeError`	for any other type (including `None`).
`ValueError`	for a score outside `[0, 1]` or `NaN`, or an invalid mapping.

output_text ¶

output_text(trace: Trace, *, path: OutputPath | None = None) -> str | None

The text that contains/regex search: to_text of output_value.

None when the (selected) output is None.

Raises:

Type	Description
`OutputPathError`	if `path` does not exist in the output.

output_value ¶

output_value(trace: Trace, *, path: OutputPath | None = None) -> JsonValue

The value an output evaluator checks: trace.output or the part path points to.

With a path, a string output is first parsed as JSON (agents often return JSON text). Dict keys are looked up as strings; on lists a segment must be an index ("items.0"; negative indexes count from the end).

Raises:

Type	Description
`OutputPathError`	if the path does not exist in the output.
`EvaluatorConfigError`	if `path` itself is malformed.

register_builtins ¶

register_builtins(registry: EvaluatorRegistry | None = None) -> None

Register the built-ins in registry (default: default_registry).

Called on import; call it yourself to populate a fresh EvaluatorRegistry. Existing entries with the same names are replaced.

to_text ¶

to_text(value: JsonValue) -> str | None

Text form of an output value: a string as is, None as None, anything else (dict, list, number, bool) as canonical JSON (sorted keys, ", "/": " separators, non-ASCII kept), so {"b": 1, "a": "é"} reads {"a": "é", "b": 1}.

evaluator ¶

evaluator(fn: Callable[..., Any]) -> FunctionEvaluator

evaluator(*, name: str | None = None, threshold: float = DEFAULT_THRESHOLD, register: bool = True, replace: bool = False, registry: EvaluatorRegistry | None = None) -> Callable[[Callable[..., Any]], FunctionEvaluator]

evaluator(fn: Callable[..., Any] | None = None, /, *, name: str | None = None, threshold: float = DEFAULT_THRESHOLD, register: bool = True, replace: bool = False, registry: EvaluatorRegistry | None = None) -> FunctionEvaluator | Callable[[Callable[..., Any]], FunctionEvaluator]

Turn fn(trace, case, **params) into a registered FunctionEvaluator.

Usable bare or with options:

@agentlens.evaluator
def mentions_order(trace, case) -> bool:
    return "order" in str(trace.output)

@agentlens.evaluator(name="short_answer", threshold=0.8)
def short(trace, case, max_words: int = 50) -> tuple[float, str]:
    words = len(str(trace.output).split())
    return min(1.0, max_words / max(words, 1)), f"{words} words"

The function may return an EvalResult, a bool, a score in [0, 1] (passes when >= threshold), (score_or_bool, reason) or a mapping with score/passed (see to_eval_result). Exceptions become failed results instead of propagating.

Parameters after (trace, case) are filled from the suite spec: {short_answer: {max_words: 20}} or, through the scalar shorthand, {short_answer: 20} (the first such parameter; threshold if there is none).

Parameters:

Name	Type	Description	Default
`name`	`str \| None`	Registry/result name; defaults to the function name.	`None`
`threshold`	`float`	Pass threshold for bare float scores.	`DEFAULT_THRESHOLD`
`register`	`bool`	Add it to the registry so suites can refer to it by name.	`True`
`replace`	`bool`	Override an existing evaluator with the same name. Re-defining the same function (a re-run notebook cell) replaces it anyway.	`False`
`registry`	`EvaluatorRegistry \| None`	Registry to use instead of `default_registry`.	`None`

Raises:

Type	Description
`TypeError`	if `fn` does not accept `(trace, case)` positionally.
`DuplicateEvaluatorError`	if the name is taken by another evaluator.

judge_cache_key ¶

judge_cache_key(judge_model: str, prompt: str, output: str) -> str

SHA-256 hex digest of (judge model, rendered prompt, output text) (docs/design.md §5).

The three parts are JSON-encoded as a list before hashing, so no choice of strings can make two different triples collide by concatenation.

available_evaluators ¶

available_evaluators() -> list[str]

Names registered in the default_registry, sorted.

get_evaluator ¶

get_evaluator(spec: SpecLike, /, **params: JsonValue) -> Evaluator

Build an evaluator from the default_registry (see EvaluatorRegistry.get).

register ¶

register(name: str, factory: EvaluatorFactory, *, primary: str | None = None, replace: bool = False) -> RegisteredEvaluator

Register factory in the default_registry.

See EvaluatorRegistry.register.

unregister ¶

unregister(name: str) -> None

Remove name from the default_registry.

aevaluate_case `async` ¶

aevaluate_case(trace: Trace, case: Case, evaluators: Sequence[Evaluator | SpecLike] | None = None, *, registry: EvaluatorRegistry | None = None) -> list[EvalResult]

Async evaluate_case: evaluators run concurrently, results keep their order.

arun_evaluator `async` ¶

arun_evaluator(evaluator: Evaluator, trace: Trace, case: Case) -> EvalResult

Async run_evaluator.

Awaits evaluator.aevaluate(trace, case) when the evaluator has one (LLM judges); otherwise runs the sync evaluate in a worker thread so slow evaluators do not block the event loop. Never raises.

evaluate_case ¶

evaluate_case(trace: Trace, case: Case, evaluators: Sequence[Evaluator | SpecLike] | None = None, *, registry: EvaluatorRegistry | None = None) -> list[EvalResult]

Score one trace with several evaluators, in order.

evaluators defaults to case.evaluators. All of them are resolved first (so a misspelt name fails before anything runs), then each runs through run_evaluator, so evaluator exceptions become failed results.

resolve_evaluators ¶

resolve_evaluators(evaluators: Iterable[Evaluator | SpecLike], *, registry: EvaluatorRegistry | None = None) -> list[Evaluator]

Turn a mix of evaluator objects and specs into evaluator objects.

Specs are built from registry (default: the process-wide one). Config errors (UnknownEvaluatorError, EvaluatorConfigError) are raised, not swallowed.

run_evaluator ¶

run_evaluator(evaluator: Evaluator, trace: Trace, case: Case) -> EvalResult

Call evaluator.evaluate(trace, case), never raising.

An exception (or a return value that is not an EvalResult) yields error_result: score 0, passed=False and the error in reason/details["error"], so one broken evaluator cannot crash a suite run. KeyboardInterrupt/SystemExit still propagate.

Evaluators¶

DEFAULT_THRESHOLD module-attribute ¶

UNSET module-attribute ¶

OutputPath module-attribute ¶

EvaluatorFactory module-attribute ¶

SpecLike module-attribute ¶

default_registry module-attribute ¶

AsyncEvaluator ¶

aevaluate ¶

Evaluator ¶

name property ¶

evaluate ¶

Contains ¶

evaluate ¶

CostBudget ¶

evaluate ¶

ExactMatch ¶

evaluate ¶

JsonSchema ¶

evaluate ¶

LatencyBudget ¶

evaluate ¶

LLMPassFail ¶

parse_reply ¶

make_result ¶

LLMRubric ¶

parse_reply ¶

make_result ¶

MaxSteps ¶

evaluate ¶

NoError ¶

evaluate ¶

OutputPathError ¶

Regex ¶

evaluate ¶

ToolSequence ¶

evaluate ¶

FunctionEvaluator ¶

bind ¶

evaluate ¶

aevaluate async ¶

DuplicateEvaluatorError ¶

EvaluatorConfigError ¶

EvaluatorError ¶

UnknownEvaluatorError ¶

DiskJudgeCache ¶

path_for ¶

get ¶

set ¶

JudgeCache ¶

get ¶

set ¶

JudgeParseError ¶

LLMJudge ¶

build_prompt abstractmethod ¶

make_result abstractmethod ¶

evaluate ¶

aevaluate async ¶

MemoryJudgeCache ¶

get ¶

set ¶

clear ¶

PairwiseJudge ¶

build_prompt ¶

parse_reply ¶

compare ¶

acompare async ¶

PairwiseResult dataclass ¶

ok property ¶

reasons property ¶

reason property ¶

combine classmethod ¶

details ¶

to_eval_result ¶

EvaluatorRegistry ¶

register ¶

unregister ¶

entry ¶

get ¶

names ¶

DEFAULT_THRESHOLD `module-attribute` ¶

UNSET `module-attribute` ¶

OutputPath `module-attribute` ¶

EvaluatorFactory `module-attribute` ¶

SpecLike `module-attribute` ¶

default_registry `module-attribute` ¶

name `property` ¶

aevaluate `async` ¶

build_prompt `abstractmethod` ¶

make_result `abstractmethod` ¶

aevaluate `async` ¶

acompare `async` ¶

PairwiseResult `dataclass` ¶

ok `property` ¶

reasons `property` ¶

reason `property` ¶

combine `classmethod` ¶

RegisteredEvaluator `dataclass` ¶

signature `instance-attribute` ¶

primary `instance-attribute` ¶

origin `instance-attribute` ¶

aevaluate_case `async` ¶

arun_evaluator `async` ¶