Reports

Reports: an experiment's results or a comparison, rendered for humans or tools.

Build a report model, then render it in a format:

from agentlens.report import ExperimentReport, render
from agentlens.suite import run_suite

result = run_suite("suites/support.yaml")
print(render(ExperimentReport.from_runs(result.experiment, result)))           # text
print(render(ExperimentReport.from_runs(result.experiment, result), "json"))   # JSON

ComparisonReport wraps a Comparison and its gate verdict; load_report reads either kind back from JSON.

Formats available today are plain text, rich (coloured tables for a terminal: render(report, "rich"), or print comparison_renderable / experiment_renderable to a rich console), json and markdown (GitHub-flavoured, for PR comments: MarkdownRenderer, starting with the hidden marker_comment) and html (one self-contained page with a span-tree trace viewer: render_html with the runs, so the traces are embedded).

DEFAULT_MARKER module-attribute

DEFAULT_MARKER = 'agentlens-report'

Key of the hidden marker: the report starts with <!-- agentlens-report -->.

HtmlRenderer

HtmlRenderer(*, runs: Iterable[RunResult] | None = None, baseline_runs: Iterable[RunResult] | None = None, title: str | None = None)

Self-contained HTML reports with a span-tree trace viewer.

Parameters:

Name	Type	Description	Default
runs	Iterable[RunResult] | None	The runs of the report's experiment (for a comparison: of the candidate), e.g. experiment_runs or a run_suite result's runs. Their traces and evaluator reasons are embedded; without runs the page shows the aggregates only.	None
baseline_runs	Iterable[RunResult] | None	The baseline's runs (comparisons only).	None
title	str | None	Page title (default: built from the experiment names).	None

comparison

comparison(report: ComparisonReport) -> str

A candidate-vs-baseline comparison, cases side by side.

experiment

experiment(report: ExperimentReport) -> str

One experiment's results, every case with its runs.

MarkdownRenderer

MarkdownRenderer(*, marker: str | None = DEFAULT_MARKER, max_cases: int = DEFAULT_MAX_CASES)

GitHub-flavoured Markdown reports, compact enough for a PR comment.

Parameters:

Name	Type	Description	Default
marker	str | None	Key of the hidden <!-- key --> first line that lets a bot find and update its comment; None for no marker.	DEFAULT_MARKER
max_cases	int	Rows per case table before the rest is summarised.	DEFAULT_MAX_CASES

Raises:

Type	Description
ValueError	for an invalid marker (see marker_comment) or max_cases < 1.

comparison

comparison(report: ComparisonReport) -> str

Verdict, metrics table, gates, case changes (collapsed) and footer.

experiment

experiment(report: ExperimentReport) -> str

Verdict, metrics, gates, failing cases (collapsed) and footer.

CaseSummary

Bases: SchemaModel

All repetitions of one case of an experiment.

errors class-attribute instance-attribute

errors: int = Field(ge=0)

Runs that raised or timed out.

scores class-attribute instance-attribute

scores: dict[str, float] = Field(default_factory=dict)

Mean score per evaluator over the repetitions.

failed_evaluators class-attribute instance-attribute

failed_evaluators: list[str] = Field(default_factory=list)

Evaluators that failed at least one repetition (sorted).

ComparisonReport

Bases: SchemaModel

A comparison and, when gates were checked, their verdict.

This is the file agentlens compare --json writes and agentlens report reads back.

ExperimentReport

Bases: SchemaModel

One experiment's results: totals, aggregate metrics, per-case summaries.

pass_rate instance-attribute

pass_rate: float

Fraction of runs that passed (metrics["pass_rate"] averages per case first).

metrics class-attribute instance-attribute

metrics: dict[str, float] = Field(default_factory=dict)

run_metrics means: what absolute gates check.

cases class-attribute instance-attribute

cases: list[CaseSummary] = Field(default_factory=list)

In the order the cases first appear in the runs.

from_runs classmethod

from_runs(experiment: Experiment, runs: ExperimentRun | Iterable[RunResult], *, gates: GateReport | None = None) -> ExperimentReport

Summarise runs (a run_suite result or stored runs).

ReportError

Bases: ValueError

A report file could not be read or is not a report AgentLens wrote.

JSONRenderer

The report model as indented JSON (readable back with load_report).

experiment

experiment(report: ExperimentReport) -> str

ExperimentReport JSON.

comparison

comparison(report: ComparisonReport) -> str

ComparisonReport JSON.

PlainTextRenderer

Plain-text reports (the CLI's default output).

experiment

experiment(report: ExperimentReport) -> str

Header, totals, aggregate metrics, per-case table and gates.

comparison

comparison(report: ComparisonReport) -> str

Header, metric deltas with CIs, case regressions/improvements and gates.

Renderer

Bases: Protocol

Renders both kinds of report in one format.

experiment

experiment(report: ExperimentReport) -> str

One experiment's results.

comparison

comparison(report: ComparisonReport) -> str

A candidate-vs-baseline comparison.

ReportFormat

Bases: StrEnum

Output formats of agentlens report.

RICH class-attribute instance-attribute

RICH = 'rich'

Coloured tables for a terminal (RichRenderer).

MARKDOWN class-attribute instance-attribute

MARKDOWN = 'markdown'

GitHub-flavoured Markdown for PR comments (MarkdownRenderer).

HTML class-attribute instance-attribute

HTML = 'html'

Self-contained page with a trace viewer (HtmlRenderer).

ReportFormatUnavailableError

Bases: LookupError

No renderer is registered for the requested format (yet).

RichRenderer

RichRenderer(*, width: int | None = None, color: bool = True)

Renders reports with rich into a string (ANSI colours unless disabled).

Parameters:

Name	Type	Description	Default
width	int | None	Line width; default the terminal's width ($COLUMNS), else DEFAULT_WIDTH.	None
color	bool	False for the same layout without colours. Rich also honours $NO_COLOR.	True

experiment

experiment(report: ExperimentReport) -> str

One experiment's results (see experiment_renderable).

comparison

comparison(report: ComparisonReport) -> str

A comparison (see comparison_renderable).

render_html

render_html(report: ExperimentReport | ComparisonReport, *, runs: Iterable[RunResult] | None = None, baseline_runs: Iterable[RunResult] | None = None, title: str | None = None) -> str

Render report as a self-contained HTML page (see HtmlRenderer).

marker_comment

marker_comment(key: str = DEFAULT_MARKER) -> str

The hidden HTML comment <!-- key --> that identifies a report.

Raises:

Type	Description
ValueError	if key is empty or has characters other than letters, digits and _ . : / @ + = - (so it cannot close the comment).

load_report

load_report(path: str | PathLike[str]) -> ExperimentReport | ComparisonReport

Read a report JSON file.

Accepts a ComparisonReport (agentlens compare --json), an ExperimentReport (agentlens report --format json) or a bare Comparison (comparison.model_dump_json()).

Raises:

Type	Description
ReportError	if the file cannot be read or is none of those.

fmt_value

fmt_value(value: float | None, *, signed: bool = False) -> str

A number with 4 significant digits (- for None), + sign if signed.

table

table(headers: Sequence[str], rows: Sequence[Sequence[str]], *, indent: str = '') -> str

Left-aligned columns separated by two spaces (no trailing spaces).

available_formats

available_formats() -> list[ReportFormat]

Formats that have a renderer, in declaration order.

get_renderer

get_renderer(fmt: ReportFormat | str) -> Renderer

The renderer for fmt.

Raises:

Type	Description
ReportFormatUnavailableError	if no renderer is registered for it.
ValueError	if fmt is not a ReportFormat.

register_renderer

register_renderer(fmt: ReportFormat | str, renderer: Renderer) -> None

Make renderer the one used for fmt (replacing any previous one).

comparison_renderable

comparison_renderable(report: ComparisonReport) -> RenderableType

A comparison: header, metric deltas with CIs, per-case changes and gates.

experiment_renderable

experiment_renderable(report: ExperimentReport) -> RenderableType

One experiment's results: header, metrics, failing cases and gates.