Schema

Core data model of AgentLens (docs/design.md §1).

Every other package speaks in these types: the tracer produces Trace objects, the store persists them, evaluators turn them into EvalResult and the suite runner groups everything into an Experiment.

Case

Bases: SchemaModel

One dataset row: an input to feed the agent and what we expect back.

evaluators class-attribute instance-attribute

evaluators: list[EvaluatorSpec] = Field(default_factory=list)

Evaluators for this case, applied on top of the suite defaults.

EvaluatorSpec

Bases: SchemaModel

A reference to an evaluator plus its parameters.

Accepts the three spellings used in suite YAML (see docs/design.md §6):

no_error                          # bare name
{max_steps: 8}                    # name -> single scalar parameter
{tool_sequence: {mode: ordered}}  # name -> parameter mapping

as well as the canonical {name: ..., params: {...}} form. A scalar parameter is stored as params={"value": 8}. Specs serialise back to the shortest YAML-style form, so a suite file round-trips unchanged.

EvalResult

Bases: SchemaModel

The verdict of one evaluator on one trace.

score class-attribute instance-attribute

score: float = Field(ge=0.0, le=1.0)

Normalised to [0, 1] so different evaluators can be averaged and compared (a 1-5 rubric score of 4 becomes 0.75).

Experiment

Bases: SchemaModel

One execution of a suite: the unit that gets compared against a baseline.

config class-attribute instance-attribute

config: dict[str, JsonValue] = Field(default_factory=dict)

What was run: model, prompt version, git sha, ...

Span

Bases: SchemaModel

One timed unit of work inside a trace: an LLM call, a tool call, a step...

Spans form a tree through parent_id; root spans have parent_id=None.

name instance-attribute

name: str

For TOOL_CALL spans this is the tool name; for LLM_CALL usually the model.

end_time class-attribute instance-attribute

end_time: UtcDatetime | None = None

None while the span is still running.

attributes class-attribute instance-attribute

attributes: dict[str, JsonValue] = Field(default_factory=dict)

Free-form metadata: model, temperature, tool arguments schema, ...

latency_ms property

latency_ms: float | None

Wall-clock duration in milliseconds, or None if the span is unfinished.

SpanKind

Bases: StrEnum

What a span represents (FR-T2).

Trace

Bases: SchemaModel

A complete agent run: its input, final output and the tree of spans.

Derived metrics (total_cost, total_tokens, latency_ms, tool_calls, n_steps) are plain properties, not serialised fields: they are always recomputed from the spans, so a stored trace can never hold stale totals.

error class-attribute instance-attribute

error: str | None = None

Set when the agent raised; the run is still recorded (FR-T5).

metadata class-attribute instance-attribute

metadata: dict[str, JsonValue] = Field(default_factory=dict)

Run context: git sha, agent version, case id, repetition index, ...

total_tokens property

total_tokens: int

Sum of input and output tokens over all spans with usage.

total_cost property

total_cost: float | None

Sum of known span costs in USD.

None when no span carries a cost (unknown is not the same as free).

latency_ms property

latency_ms: float | None

Wall-clock duration of the whole run, or None if it has not ended.

tool_calls property

tool_calls: list[str]

Names of the tools called, in call order (used by tool_sequence).

n_steps property

n_steps: int

Number of agent steps.

Counts AGENT_STEP spans when the agent records them; otherwise falls back to the number of LLM calls, since each LLM call is one turn of the agent loop.

roots

roots() -> list[Span]

Spans without a parent, in start-time order.

children

children(span_id: str) -> list[Span]

Direct children of span_id, in start-time order.

spans_of

spans_of(kind: SpanKind) -> list[Span]

All spans of a given kind, in start-time order.

Usage

Bases: SchemaModel

Token usage and estimated cost of an LLM call.

cost_usd class-attribute instance-attribute

cost_usd: NonNegativeFloat | None = None

Estimated cost; None when the model is missing from the pricing table.

total_tokens property

total_tokens: int

Input plus output tokens.

id_datetime

id_datetime(id_: str) -> datetime

Return the creation time (UTC, millisecond precision) embedded in a UUIDv7 id.

Raises:

Type	Description
ValueError	if id_ is not a valid UUID version 7.

new_id

new_id() -> str

Return a new UUIDv7 as a canonical 36-character string.

Ids are unique and monotonically increasing within a process, and roughly time-ordered across processes.