Comparison & gates¶
Compare a candidate experiment against a baseline (docs/design.md §7, ADR 0005).
Per case and in aggregate: pass rate, mean score per evaluator, cost, latency and steps, each case averaged over its repetitions, with a paired bootstrap confidence interval (10 000 resamples, seeded) on every aggregate delta. A metric is a regression only when its CI lies entirely on the bad side of 0, so noise does not make a gate flap:
from agentlens.compare import compare_experiments
from agentlens.store import SQLiteStore
with SQLiteStore("runs.db") as store:
comparison = compare_experiments(store, "main", "pr-123")
print([m.name for m in comparison.regressions])
print([c.case_id for c in comparison.case_regressions])
print(comparison.model_dump_json(indent=2))
compare_runs does the same on in-memory results (two
ExperimentRun), and paired_bootstrap is the
statistic on its own.
Gates (FR-S5) turn a comparison, or a single run, into a pass/fail verdict and a process exit code:
from agentlens.compare import evaluate_gates, load_gates
report = evaluate_gates(load_gates("gates.yaml"), comparison=comparison)
print(report.summary())
raise SystemExit(report.exit_code) # 0 pass, 1 a gate failed
DEFAULT_CONFIDENCE
module-attribute
¶
Default confidence level of the interval.
DEFAULT_RESAMPLES
module-attribute
¶
Bootstrap resamples used by default (ADR 0005).
LOWER_IS_BETTER
module-attribute
¶
Metrics where a decrease is an improvement; every other metric is higher-is-better.
SCORE_PREFIX
module-attribute
¶
Metric name of an evaluator's mean score: score.<evaluator>.
CASE_REGRESSIONS
module-attribute
¶
Gate on the number of cases whose pass rate dropped.
COUNT_GATES
module-attribute
¶
COUNT_GATES = frozenset({REGRESSIONS, CASE_REGRESSIONS})
Gates on counts taken from a Comparison; only min/max apply.
DEFAULT_MIN_CASES
module-attribute
¶
Below this many cases the report carries a (non-failing) note, after ADR 0005.
EXIT_GATE_FAILED
module-attribute
¶
Process exit code when at least one gate failed.
METRIC_ALIASES
module-attribute
¶
METRIC_ALIASES: dict[str, str] = {'success_rate': PASS_RATE, 'cost': 'cost_usd', 'latency': 'latency_ms'}
Friendlier names accepted in a gates file, mapped to comparison metric names.
REGRESSIONS
module-attribute
¶
Gate on the number of aggregate metrics that regressed (CI on the bad side).
OnMissing
module-attribute
¶
What a gate does when its metric has no data: fail (default) or be skipped.
GateReport ¶
Bases: SchemaModel
Every gate's verdict; exit_code is what a CI step should exit with.
notes
class-attribute
instance-attribute
¶
Things worth knowing that never fail the report (e.g. too few cases).
GateResult ¶
Bases: SchemaModel
The verdict of one bound of one gate (success_rate: {min: 0.9}).
actual
class-attribute
instance-attribute
¶
The value checked (candidate value, delta, percent change or count); None if unknown.
passed
instance-attribute
¶
True for skipped gates too: only a failed gate fails the report.
skipped
class-attribute
instance-attribute
¶
The gate could not be checked and was not counted as a failure.
GateRule ¶
Bases: SchemaModel
The bounds one metric must respect; at least one is required.
min/max apply to the candidate's value, the *_delta bounds to
candidate - baseline and the *_delta_pct bounds to that change in
percent of the baseline. Every bound is inclusive.
Gates ¶
Bases: SchemaModel
A set of gates, usually loaded with load_gates.
rules is keyed by the name as written (success_rate); see
resolve_metric for the metric it refers to.
min_cases
class-attribute
instance-attribute
¶
min_cases: int = Field(default=DEFAULT_MIN_CASES, ge=0)
Fewer cases than this adds a note to the report (never a failure); 0 disables it.
on_missing
class-attribute
instance-attribute
¶
on_missing: OnMissing = 'fail'
Default for gates whose metric has no data (see GateRule).
from_dict
classmethod
¶
from_dict(data: Any) -> Gates
Build gates from the YAML shape ({metric: {rule: value}, options: {...}}).
Raises:
| Type | Description |
|---|---|
GatesError
|
listing every problem, each with its location. |
evaluate ¶
evaluate(comparison: Comparison | None = None, run: ExperimentRun | Iterable[RunResult] | None = None) -> GateReport
Shortcut for evaluate_gates.
GatesError ¶
GatesError(message: str, issues: Sequence[SuiteIssue] = ())
Bases: SuiteError
A gates file could not be loaded; every problem is in issues.
A SuiteError, so tools can report suite and gates
configuration errors the same way (file:line:column: path: message).
CaseComparison ¶
Bases: SchemaModel
One case present in both experiments.
status follows the case's pass rate: regression when it dropped
(e.g. passed in the baseline, fails in the candidate), improvement
when it rose. With repetitions a single flaky run is enough to change
it, so read it as "where to look"; whether the change is beyond noise
is what the aggregate MetricDelta CIs say.
metrics
class-attribute
instance-attribute
¶
metrics: dict[str, CaseMetric] = Field(default_factory=dict)
Keyed by metric name (as in MetricDelta.name).
regressed_evaluators
class-attribute
instance-attribute
¶
Evaluators whose pass rate on this case dropped (sorted).
CaseMetric ¶
Bases: SchemaModel
One metric of one case, each side averaged over its repetitions.
delta
class-attribute
instance-attribute
¶
candidate - baseline when both are known.
Comparison ¶
Bases: SchemaModel
What compare_experiments returns: candidate vs baseline.
metrics
class-attribute
instance-attribute
¶
metrics: list[MetricDelta] = Field(default_factory=list)
pass_rate, score.<evaluator> (by name), cost_usd, latency_ms, steps.
cases
class-attribute
instance-attribute
¶
cases: list[CaseComparison] = Field(default_factory=list)
Cases present in both experiments, in baseline order.
added_cases
class-attribute
instance-attribute
¶
Case ids only in the candidate (not part of any delta).
removed_cases
class-attribute
instance-attribute
¶
Case ids only in the baseline (not part of any delta).
regressions
property
¶
regressions: list[MetricDelta]
Metrics whose CI lies entirely on the bad side.
improvements
property
¶
improvements: list[MetricDelta]
Metrics whose CI lies entirely on the good side.
case ¶
case(case_id: str) -> CaseComparison
The comparison of case case_id (KeyError if not in both experiments).
Direction ¶
MetricDelta ¶
Bases: SchemaModel
Aggregate change of one metric over the cases present in both experiments.
baseline/candidate are means over the paired cases (each case
first averaged over its repetitions), so delta == candidate - baseline
is the mean paired difference whose bootstrap CI is [ci_low, ci_high].
Status ¶
Bases: StrEnum
Verdict on one metric (aggregate) or one case.
NO_DATA
class-attribute
instance-attribute
¶
No case has a value in both experiments (e.g. no costs recorded).
paired_bootstrap ¶
paired_bootstrap(diffs: Sequence[float], *, resamples: int = DEFAULT_RESAMPLES, confidence: float = DEFAULT_CONFIDENCE, seed: int | None = 0) -> tuple[float, float]
Percentile bootstrap confidence interval of the mean of paired differences.
Draws resamples resamples of len(diffs) differences with
replacement (random.Random(seed)), takes each resample's mean and
returns the (1 - confidence) / 2 and (1 + confidence) / 2
percentiles of those means.
Which cases are drawn depends only on seed, len(diffs) and
resamples — never on the values — so the same seed gives the same
interval, and adding a constant to every difference shifts the interval
by that constant. The interval is widened, if needed, to contain the
observed mean (this only matters for floating-point rounding, e.g. when
every difference is the same).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
diffs
|
Sequence[float]
|
One difference per case ( |
required |
resamples
|
int
|
Number of bootstrap resamples (>= 1). |
DEFAULT_RESAMPLES
|
confidence
|
float
|
Confidence level, strictly between 0 and 1. |
DEFAULT_CONFIDENCE
|
seed
|
int | None
|
Seed of the random generator; |
0
|
Returns:
| Type | Description |
|---|---|
tuple[float, float]
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
if |
percentile ¶
The q-quantile (0 <= q <= 1) of sorted_values, linearly interpolated.
Same definition as NumPy's default (method="linear"): position
q * (n - 1) between the two nearest order statistics.
Raises:
| Type | Description |
|---|---|
ValueError
|
if |
classify ¶
Regression/improvement when the CI [ci_low, ci_high] excludes 0, else unchanged.
compare_experiments ¶
compare_experiments(store: Store, baseline: str | Experiment, candidate: str | Experiment, *, resamples: int = DEFAULT_RESAMPLES, seed: int | None = 0, confidence: float = DEFAULT_CONFIDENCE) -> Comparison
Compare two stored experiments (each an id, a name or an Experiment).
Loads both with experiment_runs and calls compare_runs;
see there for the arguments. A name resolves to the most recent
experiment with that name (find_experiment).
Raises:
| Type | Description |
|---|---|
ExperimentNotFoundError
|
if either experiment is not stored. |
ValueError
|
for |
compare_runs ¶
compare_runs(baseline: ExperimentRun | Iterable[RunResult], candidate: ExperimentRun | Iterable[RunResult], *, resamples: int = DEFAULT_RESAMPLES, seed: int | None = 0, confidence: float = DEFAULT_CONFIDENCE, baseline_experiment: Experiment | None = None, candidate_experiment: Experiment | None = None) -> Comparison
Compare two sets of in-memory runs (e.g. two run_suite results).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
baseline
|
ExperimentRun | Iterable[RunResult]
|
The reference runs: an |
required |
candidate
|
ExperimentRun | Iterable[RunResult]
|
The runs under test, same forms. |
required |
resamples
|
int
|
Bootstrap resamples per metric (ADR 0005: 10 000). |
DEFAULT_RESAMPLES
|
seed
|
int | None
|
Bootstrap seed, so the same inputs give the same CIs; |
0
|
confidence
|
float
|
Confidence level of the CIs. |
DEFAULT_CONFIDENCE
|
baseline_experiment
|
Experiment | None
|
Recorded on the result (default: the
|
None
|
candidate_experiment
|
Experiment | None
|
Same, for the candidate. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
A |
Comparison
|
class: |
a |
Comparison
|
class: |
Raises:
| Type | Description |
|---|---|
ValueError
|
for |
experiment_runs ¶
experiment_runs(store: Store, experiment: str | Experiment) -> list[RunResult]
Rebuild the runs of a stored experiment (traces + their eval results).
Each trace's case_id and repetition come from its metadata, as
recorded by run_suite. Traces without a string
case_id (saved under the experiment by other means) are skipped with
a warning. Runs are returned oldest first.
Raises:
| Type | Description |
|---|---|
ExperimentNotFoundError
|
if the experiment is not stored. |
metric_direction ¶
metric_direction(name: str) -> Direction
Whether metric name should go up (pass rate, scores) or down (cost, latency, steps).
run_metrics ¶
run_metrics(runs: ExperimentRun | Iterable[RunResult]) -> dict[str, tuple[float, int]]
Absolute metrics of one experiment, computed like the comparison's.
Each case is first averaged over its repetitions, then each metric is the mean over the cases that have a value for it. Used to gate a single run without a baseline.
Returns:
| Type | Description |
|---|---|
dict[str, tuple[float, int]]
|
|
dict[str, tuple[float, int]]
|
the comparison's metric order; |
dict[str, tuple[float, int]]
|
when no run has a value. |
evaluate_gates ¶
evaluate_gates(gates: Gates, comparison: Comparison | None = None, run: ExperimentRun | Iterable[RunResult] | None = None) -> GateReport
Check every gate and return the verdict.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
gates
|
Gates
|
The gates (see |
required |
comparison
|
Comparison | None
|
Candidate vs baseline. Needed by delta rules and by the
|
None
|
run
|
ExperimentRun | Iterable[RunResult] | None
|
The candidate's runs. When given, |
None
|
A gate whose metric has no data (no costs recorded, an evaluator that did
not run) fails with the reason, unless its on_missing is skip.
Raises:
| Type | Description |
|---|---|
ValueError
|
if neither |
load_gates ¶
load_gates(path: str | PathLike[str]) -> Gates
Load a gates YAML file.
Raises:
| Type | Description |
|---|---|
GatesError
|
if the file is missing, not valid YAML or not valid gates,
listing every problem at its |
resolve_metric ¶
The comparison metric a gate name refers to (aliases resolved).
Raises:
| Type | Description |
|---|---|
ValueError
|
for an unknown name, with a "did you mean" hint. |