Replay

Deterministic replay of agent runs with cassettes (docs/design.md §4, ADR 0004).

Record the LLM and tool calls of a run once, then play them back without the network, so a change to the agent's code can be tested in isolation:

from agentlens.replay import Cassette

cassette = Cassette()                                   # mode="record"
llm = agentlens.wrap_llm(cassette.wrap_llm(real_client))
lookup = agentlens.wrap_tool(cassette.wrap_tool(lookup_order))
...
cassette.save("cassettes/refund.jsonl")

cassette = Cassette.load("cassettes/refund.jsonl")      # mode="strict"
llm = agentlens.wrap_llm(cassette.wrap_llm())           # no real client needed

Modes: strict (index and request key must match, else CassetteMissError), lenient (index only), fallthrough (a miss calls the real client, and with record_new=True records it).

Where a replay departs from the recording is reported as a ReplayDivergence (call index + unified request diff): on the strict error (exc.divergence), in cassette.divergences / cassette.first_divergence in every mode, on the trace (replay_diverged span attribute, replay_divergence trace metadata), and by find_divergence for two recorded traces.

CassetteMode module-attribute

CassetteMode = Literal['record', 'strict', 'lenient', 'fallthrough']

How a cassette answers calls; see agentlens.replay.cassette.

EntryKind module-attribute

EntryKind = Literal['llm', 'tool']

The two kinds of recorded calls. Each kind has its own index sequence.

CASSETTE_FORMAT module-attribute

CASSETTE_FORMAT: Final = 'agentlens-cassette'

Value of format in the header line of a cassette file.

CASSETTE_VERSION module-attribute

CASSETTE_VERSION: Final = 1

Current cassette file format version.

DIVERGED_ATTR module-attribute

DIVERGED_ATTR: Final = 'replay_diverged'

Span attribute set to True on a replayed call that diverged.

DIVERGENCE_ATTR module-attribute

DIVERGENCE_ATTR: Final = 'replay_divergence'

Span attribute with the divergence summary; also the trace metadata key holding the first divergence's to_dict (without the full requests).

DIFF_MAX_LINE_CHARS module-attribute

DIFF_MAX_LINE_CHARS: Final = 500

Default maximum length of one diff line (longer lines are cut with ...).

DIFF_MAX_LINES module-attribute

DIFF_MAX_LINES: Final = 200

Default maximum number of lines kept in a divergence diff.

DivergenceReason module-attribute

DivergenceReason = Literal['request_changed', 'tool_changed', 'extra_call', 'missing_call']

Why a call diverged:

• request_changed: same call (and tool name), different request;
• tool_changed: the replay called another tool than the recording at this index;
• extra_call: the replay made a call the recording does not have;
• missing_call: the recording has a call the replay did not make (only detectable once the replay is over, e.g. by find_divergence).

Cassette

Cassette(entries: Iterable[CassetteEntry] = (), *, mode: CassetteMode = 'record', record_new: bool = False, metadata: dict[str, JsonValue] | None = None)

Recorded LLM and tool calls, with record and playback modes.

Wrap clients with wrap_llm and tools with wrap_tool; all wrappers of one cassette share its call counters. Wrap inside the tracing wrappers (agentlens.wrap_llm(cassette.wrap_llm(client))) so played-back calls are traced exactly like live ones. Errors raised by the real client or tool are not recorded (the call does not consume an index). Calls made from inside a real tool execution are passed straight through.

Calls are indexed in the order they reach the cassette; concurrent calls through one cassette are safe but their order (and so their index) is only as deterministic as the agent's scheduling.

Parameters:

Name	Type	Description	Default
entries	Iterable[CassetteEntry]	Initial entries (e.g. from a file); (kind, index) must be unique.	()
mode	CassetteMode	"record" (default), "strict", "lenient" or "fallthrough". It can be changed later through the mode attribute.	'record'
record_new	bool	In fallthrough mode, store calls answered by the real client/tool (replacing a mismatching entry at that index).	False
metadata	dict[str, JsonValue] | None	Free JSON metadata saved in the file header.	None

Raises:

Type	Description
ValueError	for an unknown mode.
CassetteFormatError	if two entries share a (kind, index).

mode property writable

mode: CassetteMode

The current mode; assign to switch (e.g. from "record" to "strict").

record_new instance-attribute

record_new = record_new

In fallthrough mode, store calls answered by the real client/tool.

metadata instance-attribute

metadata: dict[str, JsonValue] = dict(metadata or {})

Free JSON metadata (saved in the header); model/provider are used as defaults by wrap_llm during playback.

entries property

entries: list[CassetteEntry]

All entries, in the order they were recorded (or loaded).

unplayed property

unplayed: list[CassetteEntry]

Entries not reached yet by playback (useful to assert a replay was complete).

divergences property

divergences: list[ReplayDivergence]

Calls of this playback whose request differed from the recording, in call order.

Filled in every playback mode: strict (each one also raised a CassetteMissError), lenient (the recorded response was played anyway) and fallthrough (the real client/tool answered). Cleared by rewind.

first_divergence property

first_divergence: ReplayDivergence | None

The first entry of divergences, or None if the replay matched so far.

entries_of

entries_of(kind: EntryKind) -> list[CassetteEntry]

The entries of one kind, sorted by index.

rewind

rewind() -> None

Reset the call counters and divergences, to record or play again from the start.

wrap_llm

wrap_llm(client: LLMClient | None = None, *, model: str | None = None, provider: str | None = None) -> CassetteLLM

Return an LLMClient answering through this cassette.

client is the real client: required to record and for fallthrough misses, optional for strict/lenient playback. The returned client's model/provider attributes (read by agentlens.wrap_llm for span names and attributes) are the arguments, else the client's, else the cassette metadata values; the client's values are stored in metadata when it has none yet.

wrap_tool

wrap_tool(fn: Callable[P, R], /, *, name: str | None = ...) -> Callable[P, R]

wrap_tool(*, name: str | None = ...) -> Callable[[Callable[P, R]], Callable[P, R]]

wrap_tool(fn: Callable[P, R] | None = None, /, *, name: str | None = None) -> Callable[P, R] | Callable[[Callable[P, R]], Callable[P, R]]

Record/play back calls of a tool (sync or async), like wrap_llm for LLMs.

Usable as a wrapper or decorator (@cassette.wrap_tool). Calls are keyed by tool_key of the tool name (name, else fn.__name__) and the JSON of the arguments by parameter name. Playback returns the recorded return value as JSON (dicts, lists, strings, numbers), so tools returning other objects get their JSON form.

The wrapper's __name__ is the tool name, so wrapping it with agentlens.wrap_tool gives the span that name too. A function already wrapped by agentlens.wrap_tool is unwrapped, put behind the cassette and traced again, so its tool_call span is still recorded when the tool is played back.

save

save(path: str | PathLike[str]) -> int

Write the cassette as JSONL; return the number of entries written.

Line 1 is the CassetteHeader, then one CassetteEntry per line in recording order. The file is written to a temporary sibling and renamed atomically; parent directories are created.

load classmethod

load(path: str | PathLike[str], *, mode: CassetteMode = 'strict', record_new: bool = False) -> Cassette

Read a cassette written by save (default mode: strict).

The header line is optional (a hand-written file may be just entries) but, if present, must be the first non-blank line. Blank lines are skipped.

Raises:

Type	Description
CassetteFormatError	on an invalid line, an unsupported header or a duplicate (kind, index), naming the file and line.

from_trace classmethod

from_trace(trace: Trace, *, mode: CassetteMode = 'strict', record_new: bool = False) -> Cassette

Build a cassette from the llm_call and tool_call spans of a recorded trace.

Works for traces recorded with agentlens.wrap_llm / agentlens.wrap_tool (with or without a cassette): spans are taken in start order, failed or unfinished spans and spans nested inside a tool_call are skipped (mirroring what recording stores). LLM responses get the span's usage back; the provider raw payload is only available if the trace was recorded with include_raw=True. metadata records the trace id and name and, when known, the client's default model and provider.

Raises:

Type	Description
CassetteFormatError	if an llm_call span's input/output is not a valid request/response.

CassetteEntry

Bases: SchemaModel

One recorded call (one line of a cassette file).

For llm entries request is the LLMRequest JSON (default fields omitted) and response the full LLMResponse JSON (usage and raw included); both are validated. For tool entries name is the tool name, request the arguments by parameter name and response the JSON of the return value. key is the sha256 hex digest compared in strict mode.

CassetteHeader

Bases: SchemaModel

First line of a cassette file: format marker, version and free metadata.

CassetteLLM

CassetteLLM(cassette: Cassette, client: LLMClient | None = None, *, model: str | None = None, provider: str | None = None)

An LLMClient answering from a Cassette.

Created by Cassette.wrap_llm. Attributes not defined here are looked up on the real client (when there is one).

cassette instance-attribute

cassette = cassette

The cassette answering the calls.

client instance-attribute

client = client

The real client, or None for pure playback.

model instance-attribute

model: str | None = model or _str_attr(client, 'model', cassette, 'model')

Default model (for span names); see Cassette.wrap_llm.

provider instance-attribute

provider: str | None = provider or _str_attr(client, 'provider', cassette, 'provider')

Provider name (span attribute); see Cassette.wrap_llm.

complete

complete(request: LLMRequest) -> LLMResponse

Play back the recorded response, or call (and maybe record) the real client.

acomplete async

acomplete(request: LLMRequest) -> LLMResponse

Async variant of complete (calls the real client's acomplete).

ReplayDivergence

Bases: SchemaModel

One call where a replay differs from the recording.

*_request are the request JSON documents as stored in a cassette entry (an LLMRequest without defaults for LLM calls, the arguments by parameter name for tool calls); None with a None key when that side has no call at index. diff is a bounded unified diff of the two (for tool calls, of {"name", "arguments"}), see request_diff. Build one with create.

Attributes:

Name	Type	Description
kind	EntryKind	"llm" or "tool".
index	int	Call index (per kind, 0-based), as in the cassette.
reason	DivergenceReason	See DivergenceReason.
mode	CassetteMode | None	Cassette mode during playback; None for trace comparisons.
expected_name	/ actual_name	Tool names (None for LLM calls).
expected_key	/ actual_key	Request keys (sha256); None for a side without a call.
expected_request	/ actual_request	The request JSON of each side.
diff	str	The unified diff (recorded → replayed).
diff_truncated	bool	Whether diff was cut to keep it bounded.
span_id	str | None	The replayed llm_call/tool_call span, when traced.

create classmethod

create(kind: EntryKind, index: int, *, expected_request: JsonValue = None, actual_request: JsonValue = None, expected_key: str | None = None, actual_key: str | None = None, expected_name: str | None = None, actual_name: str | None = None, mode: CassetteMode | None = None, span_id: str | None = None, max_lines: int = DIFF_MAX_LINES, max_line_chars: int = DIFF_MAX_LINE_CHARS) -> ReplayDivergence

Build a divergence, deriving reason and the bounded diff.

A side whose key is None has no call at index (extra_call when the recording lacks it, missing_call when the replay does).

summary

summary() -> str

One line describing the divergence, e.g. llm call #2: request differs ....

to_dict

to_dict(*, include_requests: bool = True) -> dict[str, JsonValue]

JSON-ready dict (for reports and trace metadata).

include_requests=False leaves out the two full requests (the diff and the keys remain), which keeps the dict small; the replayed request is also the input of the span span_id.

CassetteError

Bases: Exception

Base class for every cassette error.

CassetteFormatError

CassetteFormatError(path: str | PathLike[str] | None, line: int | None, reason: str)

Bases: CassetteError, ValueError

A cassette file (or a trace given to Cassette.from_trace) is invalid.

For files, the message names the path and the 1-based line number.

CassetteMissError

CassetteMissError(*, kind: EntryKind, index: int, mode: CassetteMode, expected: CassetteEntry | None, actual_request: JsonValue, actual_key: str, actual_name: str | None = None, divergence: ReplayDivergence | None = None)

Bases: CassetteError

A call during playback has no usable recorded counterpart.

Raised in strict mode when the recorded call at this index has a different request key (or a different tool name), and in strict and lenient mode when there is no recorded call at this index at all. In fallthrough mode a miss calls the real client instead.

The message ends with a unified diff of the recorded and actual requests; divergence holds the same information as a ReplayDivergence (for reports).

Attributes:

Name	Type	Description
kind	EntryKind	"llm" or "tool".
index		The call index (per kind, 0-based) that missed.
mode	CassetteMode	The cassette mode at the time of the call.
expected		The recorded entry at index, or None if the cassette has no call at that index.
actual_request		The request JSON of the call being made (same shape as CassetteEntry.request).
actual_key		Its key (sha256 of the normalised request).
actual_name		The tool name for tool calls, None for LLM calls.
divergence		The ReplayDivergence (call index, reason, bounded request diff).

expected_request property

expected_request: JsonValue

The recorded request JSON, or None if nothing was recorded at index.

tool_key

tool_key(name: str, arguments: JsonValue) -> str

Cassette key of a tool call: sha256 of the canonical JSON of name and arguments.

arguments is the JSON of the call's arguments by parameter name (what a tool_call span records as input).

find_divergence

find_divergence(recorded: Trace, replayed: Trace, *, kind: EntryKind = 'llm') -> ReplayDivergence | None

First call of kind where replayed departs from recorded, or None.

The calls of each trace are selected and keyed exactly like Cassette.from_trace <agentlens.replay.Cassette.from_trace> does (llm_call or tool_call spans in start order; failed, unfinished and tool-nested spans skipped), then compared index by index: a different request key or tool name gives request_changed/tool_changed, a call only in replayed extra_call and a call only in recorded missing_call. span_id is the replayed span (None for a missing call). Works for any two traces, e.g. a recorded run and a live replay against another model (FR-R2).

.. code-block:: python

divergence = find_divergence(baseline_trace, new_trace)
if divergence is not None:
    print(divergence)  # summary + unified diff of the two requests

Raises:

Type	Description
CassetteFormatError	if an llm_call span is not a valid call.

request_diff

request_diff(expected: JsonValue, actual: JsonValue, *, expected_label: str = 'recorded', actual_label: str = 'replayed', present: tuple[bool, bool] = (True, True), max_lines: int = DIFF_MAX_LINES, max_line_chars: int = DIFF_MAX_LINE_CHARS) -> tuple[str, bool]

Unified diff of two request JSON documents; return (diff, truncated).

Both documents are pretty-printed as canonical JSON (sorted keys, 2-space indent) before diffing, so key order never shows up as a change. A side marked absent in present (no call at all) diffs as an empty document. The result is empty when the documents are equal; otherwise it keeps at most max_lines lines of at most max_line_chars characters, with a final ... (N more diff lines) marker when lines were dropped. truncated tells whether anything (lines or line ends) was cut.