Top level (agentlens)¶

AgentLens: regression testing and evaluation toolkit for LLM agents.

Record agent runs as traces, replay them deterministically, score them with deterministic checks and LLM-as-judge, and catch regressions in CI.

Redactor `module-attribute` ¶

Redactor = Callable[[JsonValue], JsonValue]

Takes a JSON payload and returns it with secrets masked. Must not mutate its input.

Evaluator ¶

Bases: Protocol

Anything that scores one trace of one case (docs/design.md §5).

name is what ends up in EvalResult.evaluator; it is declared as a read-only property so both a plain attribute and a @property satisfy the protocol. evaluate should return an EvalResult; exceptions are turned into failed results by run_evaluator.

name `property` ¶

name: str

Evaluator name, as used in suite files and results.

evaluate ¶

evaluate(trace: Trace, case: Case) -> EvalResult

Score trace (the agent's run on case).

SpanKind ¶

Bases: StrEnum

What a span represents (FR-T2).

Trace ¶

Bases: SchemaModel

A complete agent run: its input, final output and the tree of spans.

Derived metrics (total_cost, total_tokens, latency_ms, tool_calls, n_steps) are plain properties, not serialised fields: they are always recomputed from the spans, so a stored trace can never hold stale totals.

error `class-attribute` `instance-attribute` ¶

error: str | None = None

Set when the agent raised; the run is still recorded (FR-T5).

metadata `class-attribute` `instance-attribute` ¶

metadata: dict[str, JsonValue] = Field(default_factory=dict)

Run context: git sha, agent version, case id, repetition index, ...

total_tokens `property` ¶

total_tokens: int

Sum of input and output tokens over all spans with usage.

total_cost `property` ¶

total_cost: float | None

Sum of known span costs in USD.

None when no span carries a cost (unknown is not the same as free).

latency_ms `property` ¶

latency_ms: float | None

Wall-clock duration of the whole run, or None if it has not ended.

tool_calls `property` ¶

tool_calls: list[str]

Names of the tools called, in call order (used by tool_sequence).

n_steps `property` ¶

n_steps: int

Number of agent steps.

Counts AGENT_STEP spans when the agent records them; otherwise falls back to the number of LLM calls, since each LLM call is one turn of the agent loop.

roots ¶

roots() -> list[Span]

Spans without a parent, in start-time order.

children ¶

children(span_id: str) -> list[Span]

Direct children of span_id, in start-time order.

spans_of ¶

spans_of(kind: SpanKind) -> list[Span]

All spans of a given kind, in start-time order.

PatternRedactor ¶

PatternRedactor(patterns: Iterable[str | Pattern[str]] = DEFAULT_PATTERNS, *, keys: Iterable[str] = DEFAULT_SENSITIVE_KEYS, replacement: str = REDACTED)

Redactor masking sensitive mapping keys and secret-looking substrings.

It walks nested lists and dicts and returns a new value; the input is never mutated.

.. code-block:: python

redactor = PatternRedactor(
    patterns=(*DEFAULT_PATTERNS, r"\bINTERNAL-[0-9]{6}\b"),
    keys=DEFAULT_SENSITIVE_KEYS | {"session_id"},
)
agentlens.run("agent", sink=store.save_trace, redactor=redactor)

Parameters:

Name	Type	Description	Default
`patterns`	`Iterable[str \| Pattern[str]]`	Regular expressions matched inside every string. A group named `secret` limits masking to that group.	`DEFAULT_PATTERNS`
`keys`	`Iterable[str]`	Mapping keys whose values are always masked. Keys ending in `_api_key`, `_secret` or `_password` are masked as well.	`DEFAULT_SENSITIVE_KEYS`
`replacement`	`str`	The text that replaces each secret.	`REDACTED`

redact_text ¶

redact_text(text: str) -> str

Mask every pattern match in text.

is_sensitive_key ¶

is_sensitive_key(key: str) -> bool

Whether the value stored under key must be masked entirely.

evaluator ¶

evaluator(fn: Callable[..., Any]) -> FunctionEvaluator

evaluator(*, name: str | None = None, threshold: float = DEFAULT_THRESHOLD, register: bool = True, replace: bool = False, registry: EvaluatorRegistry | None = None) -> Callable[[Callable[..., Any]], FunctionEvaluator]

evaluator(fn: Callable[..., Any] | None = None, /, *, name: str | None = None, threshold: float = DEFAULT_THRESHOLD, register: bool = True, replace: bool = False, registry: EvaluatorRegistry | None = None) -> FunctionEvaluator | Callable[[Callable[..., Any]], FunctionEvaluator]

Turn fn(trace, case, **params) into a registered FunctionEvaluator.

Usable bare or with options:

@agentlens.evaluator
def mentions_order(trace, case) -> bool:
    return "order" in str(trace.output)

@agentlens.evaluator(name="short_answer", threshold=0.8)
def short(trace, case, max_words: int = 50) -> tuple[float, str]:
    words = len(str(trace.output).split())
    return min(1.0, max_words / max(words, 1)), f"{words} words"

The function may return an EvalResult, a bool, a score in [0, 1] (passes when >= threshold), (score_or_bool, reason) or a mapping with score/passed (see to_eval_result). Exceptions become failed results instead of propagating.

Parameters after (trace, case) are filled from the suite spec: {short_answer: {max_words: 20}} or, through the scalar shorthand, {short_answer: 20} (the first such parameter; threshold if there is none).

Parameters:

Name	Type	Description	Default
`name`	`str \| None`	Registry/result name; defaults to the function name.	`None`
`threshold`	`float`	Pass threshold for bare float scores.	`DEFAULT_THRESHOLD`
`register`	`bool`	Add it to the registry so suites can refer to it by name.	`True`
`replace`	`bool`	Override an existing evaluator with the same name. Re-defining the same function (a re-run notebook cell) replaces it anyway.	`False`
`registry`	`EvaluatorRegistry \| None`	Registry to use instead of `default_registry`.	`None`

Raises:

Type	Description
`TypeError`	if `fn` does not accept `(trace, case)` positionally.
`DuplicateEvaluatorError`	if the name is taken by another evaluator.

get_client ¶

get_client(spec: str | None = None, *, pricing: Pricing | str | PathLike[str] | None = None, **overrides: Any) -> LLMClient

Build an LLM client from "<provider>:<model>" (default: $AGENTLENS_MODEL).

Providers and the environment variables they read (overrides win):

openai:<model> — OpenAICompatClient; OPENAI_API_KEY, OPENAI_BASE_URL (default https://api.openai.com/v1).
anthropic:<model> — AnthropicClient; ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL.
ollama:<model> — OLLAMA_BASE_URL (default http://localhost:11434/v1), no key.
vllm:<model> — VLLM_BASE_URL (default http://localhost:8000/v1), optional VLLM_API_KEY.
lmstudio:<model> — LMSTUDIO_BASE_URL (default http://localhost:1234/v1).
openai-compat:<model> — any other compatible server: OPENAI_COMPAT_BASE_URL (required, or base_url=), optional OPENAI_COMPAT_API_KEY.
mock / mock:<name> — MockLLM reporting model <name> (default "mock") that echoes the last user message; pass script= (and cycle=, default=) to script it. Costs 0 and needs no network.

Local servers never receive OPENAI_API_KEY. Every client except the mock is wrapped in a PricedClient, which fills usage.cost_usd from pricing: a Pricing, a YAML file merged over the packaged table, or None for Pricing.load (packaged table + $AGENTLENS_PRICING).

Raises:

Type	Description
`LLMConfigError`	No spec and no `AGENTLENS_MODEL`, an invalid spec, an unknown provider, a missing required setting or an invalid pricing file.

current_span ¶

current_span() -> Span | None

The innermost open span in this context, or None.

current_trace ¶

current_trace() -> Trace | None

The trace being recorded in this context, or None outside run.

redact_trace ¶

redact_trace(trace: Trace, redactor: Redactor | None = None) -> Trace

Return a copy of trace with every payload passed through redactor.

Redacted fields: the trace's input, output, metadata and error and each span's input, output, attributes and error. Ids, names, timings and usage are kept as they are. The original trace is not modified.

Parameters:

Name	Type	Description	Default
`trace`	`Trace`	The trace to redact.	required
`redactor`	`Redactor \| None`	Defaults to `DEFAULT_REDACTOR`.	`None`

Raises:

Type	Description
`Exception`	whatever `redactor` raises; `agentlens.run` catches it and drops the trace rather than persisting unredacted data.

run ¶

run(name: str, *, input: object = None, metadata: Mapping[str, object] | None = None, sink: TraceSink | None = None, redactor: Redactor | None = DEFAULT_REDACTOR) -> Iterator[Trace]

Record one agent run as a Trace.

.. code-block:: python

with agentlens.run("support-agent", input=question) as trace:
    trace.output = agent(question)
print(trace.latency_ms, trace.tool_calls)

Spans created inside the block (by trace, span or the LLM wrappers) are attached to the yielded trace. When the block exits the trace gets its end_time (and error if the block raised) and a redacted copy is passed to sink; the yielded trace itself is not redacted. Exceptions from the block are re-raised after recording.

Runs may be nested; an inner run records an independent trace.

Parameters:

Name	Type	Description	Default
`name`	`str`	Trace name, e.g. the agent or suite name.	required
`input`	`object`	The agent input; converted to JSON (`repr` as a fallback).	`None`
`metadata`	`Mapping[str, object] \| None`	Run context such as git sha, case id or repetition.	`None`
`sink`	`TraceSink \| None`	Called once with the finished trace (e.g. `store.save_trace`). Errors raised by the sink are logged, not propagated.	`None`
`redactor`	`Redactor \| None`	Masks secrets in the copy given to `sink` (NFR-5); defaults to `DEFAULT_REDACTOR`. `None` disables redaction. If it raises, the error is logged and the trace is not passed to the sink.	`DEFAULT_REDACTOR`

span ¶

span(name: str, *, kind: SpanKind | str = CUSTOM, input: object = None, attributes: Mapping[str, object] | None = None) -> Iterator[Span | None]

Record a block of code as a span.

Yields the running Span (set span.output to record a result), or None when no run is active.

.. code-block:: python

with agentlens.span("search", kind="retrieval", input=query) as s:
    docs = index.search(query)
    if s is not None:
        s.output = [d.id for d in docs]

trace ¶

trace(func: Callable[P, R]) -> Callable[P, R]

trace(*, kind: SpanKind | str = ..., name: str | None = ...) -> Callable[[Callable[P, R]], Callable[P, R]]

trace(func: Callable[P, R] | None = None, /, *, kind: SpanKind | str = CUSTOM, name: str | None = None) -> Callable[P, R] | Callable[[Callable[P, R]], Callable[P, R]]

Decorator recording each call of a sync or async function as a span.

The span input is the call's arguments by parameter name (without self/cls), the output is the return value, and an exception is stored in span.error before being re-raised.

.. code-block:: python

@agentlens.trace(kind="tool_call")
def lookup_order(order_id: str) -> dict: ...

@agentlens.trace                      # kind=custom, name=function name
async def plan(question: str) -> list[str]: ...

Parameters:

Name	Type	Description	Default
`kind`	`SpanKind \| str`	The `SpanKind` (or its string value).	`CUSTOM`
`name`	`str \| None`	Span name; defaults to the function's `__qualname__`.	`None`

Raises:

Type	Description
`TypeError`	if applied to a generator function (not supported yet).
`ValueError`	if `kind` is not a valid span kind.

wrap_llm ¶

wrap_llm(client: LLMClient, *, name: str | None = None, provider: str | None = None, include_raw: bool = False) -> TracedLLM

Wrap an LLM client so every call inside run is recorded.

Each complete/acomplete call becomes an llm_call span:

name: name if given, else the requested model, else the client's model attribute, else the client class name;
input: the LLMRequest as JSON (fields left at their defaults omitted);
output: the LLMResponse as JSON, without usage (stored in span.usage) and without the provider's raw payload unless include_raw=True;
usage: the response's token counts and cost (None if unreported);
attributes: client (class path), provider (if known), model, temperature, max_tokens, n_messages, n_tools, tool_names, response_format, request_hash (cassette key, see ADR 0004) and, once answered, response_model and n_tool_calls.

Errors raised by the client are stored in span.error and re-raised. Outside a run the wrapper calls straight through. Wrapping an already wrapped client returns it unchanged.

.. code-block:: python

llm = agentlens.wrap_llm(MockLLM(["Hello!"]))
with agentlens.run("agent") as trace:
    llm.complete(LLMRequest(messages="Hi"))
assert trace.spans[0].usage is not None

Parameters:

Name	Type	Description	Default
`client`	`LLMClient`	Any object implementing `LLMClient`.	required
`name`	`str \| None`	Fixed span name instead of the model name.	`None`
`provider`	`str \| None`	`provider` attribute; defaults to the client's own `provider` attribute when it is a string.	`None`
`include_raw`	`bool`	Also record `LLMResponse.raw` in the span output.	`False`

wrap_tool ¶

wrap_tool(fn: Callable[P, R], /, *, name: str | None = ...) -> Callable[P, R]

wrap_tool(*, name: str | None = ...) -> Callable[[Callable[P, R]], Callable[P, R]]

wrap_tool(fn: Callable[P, R] | None = None, /, *, name: str | None = None) -> Callable[P, R] | Callable[[Callable[P, R]], Callable[P, R]]

Record each call of a tool (sync or async callable) as a tool_call span.

Use it as a plain wrapper or as a decorator, with or without a name:

.. code-block:: python

search = agentlens.wrap_tool(index.search, name="search_docs")

@agentlens.wrap_tool
def lookup_order(order_id: str) -> dict[str, str]: ...

@agentlens.wrap_tool(name="refund")
async def issue_refund(order_id: str, amount: float) -> bool: ...

The span name is the tool name (name, else the function's __name__), which is what Trace.tool_calls and the tool_sequence evaluator compare. The input is the call's arguments by parameter name (self/cls dropped), the output the return value; an exception is recorded in span.error and re-raised. Outside a run the tool is called directly. The returned function exposes the original as __wrapped__ and the tool name as __agentlens_tool__; wrapping it again returns it unchanged, or re-wraps the original under a new name (never two nested spans).

Raises:

Type	Description
`TypeError`	if `fn` is a generator function (not supported yet).

Top level (agentlens)¶

Redactor module-attribute ¶

Evaluator ¶

name property ¶

evaluate ¶

SpanKind ¶

Trace ¶

error class-attribute instance-attribute ¶

metadata class-attribute instance-attribute ¶

total_tokens property ¶

total_cost property ¶

latency_ms property ¶

tool_calls property ¶

n_steps property ¶

roots ¶

children ¶

spans_of ¶

PatternRedactor ¶

redact_text ¶

is_sensitive_key ¶

evaluator ¶

get_client ¶

current_span ¶

current_trace ¶

redact_trace ¶

run ¶

span ¶

trace ¶

wrap_llm ¶

wrap_tool ¶

Redactor `module-attribute` ¶

name `property` ¶

error `class-attribute` `instance-attribute` ¶

metadata `class-attribute` `instance-attribute` ¶

total_tokens `property` ¶

total_cost `property` ¶

latency_ms `property` ¶

tool_calls `property` ¶

n_steps `property` ¶