Suites

Suites: an agent, a dataset of cases and how to run them (docs/design.md §6).

Load a suite YAML file (errors point at file:line:column):

from agentlens.suite import load_suite

suite = load_suite("suites/support.yaml")
agent = suite.load_agent()            # imports "examples.support_agent:run_agent"
for case in suite.resolved_cases():   # suite defaults merged into each case
    output = agent(case.input)

or define one in Python (FR-S1):

from agentlens.suite import Suite

suite = Suite(name="support", agent=run_agent,
              cases=[{"id": "refund", "input": "refund order 1234"}]).check()

Datasets can live in their own file (cases: data/support.jsonl in the suite, or load_cases directly).

Run it (FR-S2/S3): every case repetitions times with bounded concurrency, traces and eval results saved under a new experiment:

from agentlens.store import SQLiteStore
from agentlens.suite import run_suite, set_baseline

with SQLiteStore() as store:
    result = run_suite(suite, store=store, name="main")
    print(f"{result.pass_rate:.0%}")
    set_baseline(store, result.experiment)

DATASET_SUFFIXES module-attribute

DATASET_SUFFIXES = ('.jsonl', '.yaml', '.yml', '.json')

File types accepted for an external dataset (cases: path) and by load_cases.

ProgressCallback module-attribute

ProgressCallback = Callable[[RunResult, int, int], None]

progress(result, completed, total), called after each run is saved.

AgentImportError

AgentImportError(message: str)

Bases: SuiteError

The suite's agent (or an imports module) could not be imported or is not callable.

SuiteError

SuiteError(message: str, issues: Sequence[SuiteIssue] = ())

Bases: Exception

A suite could not be loaded: all problems found are in issues.

str(error) lists every issue on its own line, e.g.:

invalid suite suites/support.yaml (2 problems):
  suites/support.yaml:12:9: cases[2].evaluators[0]: unknown evaluator 'foo'
  suites/support.yaml:20:5: cases[3].id: duplicate case id 'refund' (first: cases[0])

SuiteIssue dataclass

SuiteIssue(message: str, loc: Loc = (), file: str | None = None, line: int | None = None, column: int | None = None)

One problem in a suite, with its position in the source file when known.

loc class-attribute instance-attribute

loc: Loc = ()

Where in the suite document (cases[2].id); empty for the whole suite.

file class-attribute instance-attribute

file: str | None = None

The file the problem is in (the suite or an external dataset), if any.

line class-attribute instance-attribute

line: int | None = None

1-based line in file.

column class-attribute instance-attribute

column: int | None = None

1-based column in file.

path property

path: str

loc rendered as cases[2].evaluators[0].

ExperimentNotFoundError

Bases: LookupError

No stored experiment has the given id or name.

Suite

Bases: SchemaModel

A test suite: the agent under test, its dataset and run settings.

cases keep exactly what was declared; resolved_cases returns them with the suite defaults merged in, which is what the runner executes. Case ids must be unique (they pair cases across experiments when comparing).

name class-attribute instance-attribute

name: str = Field(min_length=1)

Suite name; experiments are grouped by it.

agent instance-attribute

agent: str | Callable[..., Any]

The agent under test: "package.module:callable" (see resolve_agent) or, for Python-defined suites, the callable itself. Called with each case input.

description class-attribute instance-attribute

description: str | None = None

Free text for humans and reports.

model class-attribute instance-attribute

model: str | None = None

The LLM under test ("openai:gpt-4o-mini"), recorded with the experiment; informational for the runner and agents that read it.

repetitions class-attribute instance-attribute

repetitions: int = Field(default=1, ge=1)

How many times each case is run (FR-S2), to measure non-determinism.

concurrency class-attribute instance-attribute

concurrency: int = Field(default=1, ge=1)

Maximum runs executing at the same time.

timeout_s class-attribute instance-attribute

timeout_s: float | None = Field(default=None, gt=0)

Per-run timeout in seconds; None means no limit.

imports class-attribute instance-attribute

imports: list[str] = Field(default_factory=list)

Modules imported before the evaluators are checked, e.g. the module that defines custom @agentlens.evaluator functions used by the cases.

metadata class-attribute instance-attribute

metadata: dict[str, JsonValue] = Field(default_factory=dict)

Anything else worth recording with the experiment.

cases class-attribute instance-attribute

cases: list[Case] = Field(min_length=1)

The dataset, as declared (see resolved_cases).

source_path property

source_path: Path | None

The YAML file this suite was loaded from (None for Python suites).

base_dir property

base_dir: Path | None

Directory of source_path: relative paths in the suite resolve here.

search_paths

search_paths() -> list[Path]

Directories appended to sys.path before importing the agent: the current working directory, then base_dir (if any).

resolved_cases

resolved_cases() -> list[Case]

The cases with defaults merged in (new objects; cases is unchanged).

Evaluators: defaults first, then the case's own; a case evaluator replaces defaults of the same name (merge_evaluators). Tags: defaults first, then the case's own, without duplicates.

case

case(case_id: str) -> Case

The resolved case with id case_id (KeyError if none).

load_agent

load_agent() -> Callable[..., Any]

Import and return the agent callable (see resolve_agent).

Raises:

Type	Description
AgentImportError	if the import path cannot be resolved.

agent_is_async

agent_is_async() -> bool

Import the agent and tell whether it must be awaited (is_async_agent).

content_hash

content_hash() -> str

A stable sha256 hex digest of what the suite runs.

Computed from the canonical JSON (sorted keys, no whitespace) of the suite with defaults merged into the cases (resolved_cases), so two suites that run the same cases hash the same even if one spells them with defaults and the other inline, and a suite whose cases live in an external dataset changes hash when that dataset changes. Purely descriptive fields (description) and run tuning that does not change results (concurrency) are excluded. A callable agent is hashed by its module:qualname.

check

check(*, registry: EvaluatorRegistry | None = None) -> Suite

Check what the model alone cannot: imports and evaluators.

Imports the imports modules, then builds every evaluator spec (defaults and per case) from registry (default: the process-wide one), so unknown names and bad parameters fail now rather than mid-run. Built evaluators are discarded; LLM judges need no API key to be built.

Returns:

Type	Description
Suite	The suite itself, for chaining.

Raises:

Type	Description
SuiteError	listing every problem found.

issues

issues(*, registry: EvaluatorRegistry | None = None) -> list[SuiteIssue]

Every problem check would raise, without raising (no file positions).

SuiteDefaults

Bases: SchemaModel

Settings applied to every case of a suite (defaults: in YAML).

evaluators class-attribute instance-attribute

evaluators: list[EvaluatorSpec] = Field(default_factory=list)

Evaluators run on every case (see merge_evaluators for overrides).

tags class-attribute instance-attribute

tags: list[str] = Field(default_factory=list)

Tags added to every case (before the case's own, without duplicates).

CaseResult dataclass

CaseResult(case: Case, runs: list[RunResult])

All repetitions of one case.

case instance-attribute

case: Case

The resolved case (suite defaults merged in).

runs instance-attribute

runs: list[RunResult]

Ordered by repetition.

case_id property

case_id: str

The case id.

passed property

passed: int

How many repetitions passed.

pass_rate property

pass_rate: float

Fraction of repetitions that passed (0 when there are none).

errors property

errors: int

How many repetitions raised or timed out.

mean_scores

mean_scores() -> dict[str, float]

Mean score per evaluator name over the repetitions.

ExperimentRun dataclass

ExperimentRun(experiment: Experiment, suite: Suite, cases: list[CaseResult])

What run_suite returns: the experiment and every run, by case.

cases instance-attribute

cases: list[CaseResult]

In suite order.

runs property

runs: list[RunResult]

Every run, in case order then repetition order.

total property

total: int

Number of runs.

passed property

passed: int

Number of runs that passed.

errors property

errors: int

Number of runs that raised or timed out.

pass_rate property

pass_rate: float

Fraction of runs that passed (0 when there are none).

case

case(case_id: str) -> CaseResult

The result of case case_id (KeyError if none).

RunResult dataclass

RunResult(case_id: str, repetition: int, trace: Trace, eval_results: list[EvalResult] = list())

One execution of one case: the stored trace and its evaluation results.

repetition instance-attribute

repetition: int

0-based repetition index.

trace instance-attribute

trace: Trace

The trace as stored (redacted).

error property

error: str | None

The agent's exception ("Type: message"), or None.

timed_out property

timed_out: bool

Whether the run was stopped by the per-run timeout.

passed property

passed: bool

No error and every evaluator passed (no evaluators: no error is enough).

RunTimeoutError

Bases: TimeoutError

An agent call exceeded the per-run timeout (recorded on the trace).

check_import_path

check_import_path(spec: str) -> tuple[str, str]

Split "pkg.mod:obj.attr" into ("pkg.mod", "obj.attr").

Raises:

Type	Description
ValueError	if spec is not module:attribute with dotted identifiers on both sides.

import_module

import_module(name: str, *, search_paths: Iterable[PathLike] = ()) -> ModuleType

Import module name, first appending search_paths to sys.path.

Raises:

Type	Description
AgentImportError	if the module does not exist, or importing it raised (the original exception is chained as __cause__).

is_async_agent

is_async_agent(agent: Callable[..., Any]) -> bool

Whether calling agent returns an awaitable that the runner must await.

True for async def functions and methods, functools.partial of them, and objects whose __call__ is async def. A plain function that happens to return a coroutine is not detected; wrap it in async def.

resolve_agent

resolve_agent(spec: str | Callable[..., Any], *, search_paths: Iterable[PathLike] = ()) -> Callable[..., Any]

Import the agent callable named by "package.module:callable".

A callable is returned unchanged, so Python-defined suites can pass the function itself. Attribute paths walk into the module: mod:Agent.run, mod:agent.respond.

Parameters:

Name	Type	Description	Default
spec	str | Callable[..., Any]	The import path, or the agent callable itself.	required
search_paths	Iterable[PathLike]	Directories appended to sys.path (when missing) before importing — see the module docstring.	()

Raises:

Type	Description
AgentImportError	for a malformed path, a missing module or attribute, a module that fails to import, or a target that is not callable.

format_loc

format_loc(loc: Loc) -> str

Render a location as cases[2].evaluators[0] (empty for the document root).

experiment_config

experiment_config(suite: Suite, *, model: str | None = None, repetitions: int | None = None, concurrency: int | None = None, timeout_s: float | None = None, extra: Mapping[str, JsonValue] | None = None) -> dict[str, JsonValue]

The Experiment.config recorded for a run of suite.

Keys: git_sha, git_dirty (git_info, run in the suite's folder), model (resolve_model), suite_hash (Suite.content_hash), agent, repetitions, concurrency, timeout_s (the effective values), agentlens_version and suite_metadata. extra (user-provided, e.g. {"prompt_version": "v2"}) is merged last and wins on a clash.

find_experiment

find_experiment(store: Store, experiment: str | Experiment) -> Experiment

Look up a stored experiment by id, else by name (the most recent with that name).

Raises:

Type	Description
ExperimentNotFoundError	if nothing matches.

get_baseline

get_baseline(store: Store, suite: str | Suite) -> Experiment | None

The baseline experiment of suite (a name or a Suite), or None.

Should several be marked (e.g. written by hand), the most recent one wins.

git_info

git_info(cwd: str | PathLike[str] | None = None) -> dict[str, JsonValue]

The current git commit of cwd (default: the process cwd).

Returns {"git_sha": <sha or None>, "git_dirty": <bool or None>}. git_dirty is true when tracked files have uncommitted changes (untracked files are ignored). Both are None outside a repository or when git is not installed; this never raises.

resolve_model

resolve_model(suite: Suite, override: str | None = None) -> str | None

The model under test: override, else suite.model, else $AGENTLENS_MODEL.

set_baseline

set_baseline(store: Store, experiment: str | Experiment) -> Experiment

Mark experiment (id, name or object) as the baseline of its suite.

Every other baseline of the same suite is unmarked first, so a suite has at most one baseline. (If the process dies between the two steps the suite is left with no baseline, never with two.)

Returns:

Type	Description
Experiment	The stored experiment, now with is_baseline=True.

Raises:

Type	Description
ExperimentNotFoundError	if the experiment is not stored.

unset_baseline

unset_baseline(store: Store, experiment: str | Experiment) -> Experiment

Remove the baseline mark from experiment (a no-op if it has none).

Raises:

Type	Description
ExperimentNotFoundError	if the experiment is not stored.

load_cases

load_cases(path: str | PathLike[str]) -> list[Case]

Load a dataset file (.jsonl, .yaml/.yml or .json list of cases).

Raises:

Type	Description
SuiteError	listing every invalid row or duplicate id, with positions.

load_suite

load_suite(path: str | PathLike[str], *, registry: EvaluatorRegistry | None = None, check: bool = True) -> Suite

Load and validate a suite YAML file.

Parameters:

Name	Type	Description	Default
path	str | PathLike[str]	The suite file.	required
registry	EvaluatorRegistry | None	Registry to check evaluator specs against (default: the process-wide one).	None
check	bool	Also run Suite.check (import imports modules and build every evaluator spec). Schema and unique-id checks always run. The agent is not imported here; see Suite.load_agent.	True

Raises:

Type	Description
SuiteError	listing every problem found, each with its file position.

merge_evaluators

merge_evaluators(defaults: Iterable[EvaluatorSpec], own: Iterable[EvaluatorSpec]) -> list[EvaluatorSpec]

Suite defaults first, then the case's own evaluators.

A case evaluator overrides every default with the same name, so a case can tighten {max_steps: 8} to {max_steps: 3} without running both. Different names accumulate. Merging is idempotent.

arun_suite async

arun_suite(suite: Suite | str | PathLike[str], *, store: Store | None = None, name: str | None = None, model: str | None = None, repetitions: int | None = None, concurrency: int | None = None, timeout_s: float | None = None, config: Mapping[str, JsonValue] | None = None, registry: EvaluatorRegistry | None = None, redactor: Redactor | None = DEFAULT_REDACTOR, progress: ProgressCallback | None = None) -> ExperimentRun

Run every case of suite repetitions times and record an experiment.

Async version of run_suite, for callers already inside an event loop.

Parameters:

Name	Type	Description	Default
suite	Suite | str | PathLike[str]	A Suite, or the path of a suite YAML file.	required
store	Store | None	Where the experiment, traces and eval results are saved (e.g. SQLiteStore); None keeps them in the returned object only.	None
name	str | None	Experiment name; default "<suite>-YYYYmmdd-HHMMSS".	None
model	str | None	Model under test, overriding suite.model/$AGENTLENS_MODEL; passed to agents that take a model parameter.	None
repetitions	int | None	Overrides suite.repetitions.	None
concurrency	int | None	Overrides suite.concurrency.	None
timeout_s	float | None	Overrides suite.timeout_s (seconds per agent call).	None
config	Mapping[str, JsonValue] | None	Extra values for Experiment.config (e.g. a prompt version); they win over the collected ones (experiment_config).	None
registry	EvaluatorRegistry | None	Evaluator registry (default: the process-wide one).	None
redactor	Redactor | None	Applied to every trace before it is evaluated and saved; None disables redaction.	DEFAULT_REDACTOR
progress	ProgressCallback | None	progress(result, completed, total) after each run; its exceptions are logged and ignored.	None

Returns:

Name	Type	Description
An	ExperimentRun	class:ExperimentRun (experiment, per-case results, pass rate).

Raises:

Type	Description
SuiteError	if the suite file is invalid or the agent cannot be imported.
EvaluatorError	if an evaluator spec cannot be built (before anything runs).
ValueError	for repetitions/concurrency < 1 or timeout_s <= 0.

default_experiment_name

default_experiment_name(suite: Suite, created_at: datetime) -> str

"<suite>-YYYYmmdd-HHMMSS" (UTC), the name used when none is given.

run_suite

run_suite(suite: Suite | str | PathLike[str], *, store: Store | None = None, name: str | None = None, model: str | None = None, repetitions: int | None = None, concurrency: int | None = None, timeout_s: float | None = None, config: Mapping[str, JsonValue] | None = None, registry: EvaluatorRegistry | None = None, redactor: Redactor | None = DEFAULT_REDACTOR, progress: ProgressCallback | None = None) -> ExperimentRun

Run suite and record an experiment (blocking; see arun_suite).

Starts its own event loop with asyncio.run, so it cannot be called from inside a running loop — use await arun_suite(...) there.