LLM clients

Provider-neutral LLM access (docs/design.md §3, ADR 0002).

One small LLMClient protocol plus request/response models, so judges, example agents and replay can swap providers (mock, OpenAI-compatible, Anthropic) without code changes:

from agentlens.llm import LLMRequest, Message, MockLLM

llm = MockLLM(["Hi! How can I help?"])
response = llm.complete(LLMRequest(messages=[Message.user("Hello")]))
assert response.content == "Hi! How can I help?"

Real providers: OpenAICompatClient (OpenAI, Ollama, vLLM, LM Studio via base_url) and AnthropicClient (Anthropic Messages API). Adapter failures raise subclasses of LLMError.

get_client builds any of them from a "provider:model" spec (or $AGENTLENS_MODEL), with usage.cost_usd filled from the Pricing table:

from agentlens.llm import get_client

llm = get_client("ollama:llama3.1")  # or "openai:gpt-4o-mini", "anthropic:...", "mock"

ANTHROPIC_VERSION module-attribute

ANTHROPIC_VERSION = '2023-06-01'

Value of the anthropic-version header sent with every request.

DEFAULT_ANTHROPIC_BASE_URL module-attribute

DEFAULT_ANTHROPIC_BASE_URL = 'https://api.anthropic.com'

The base_url of the official Anthropic API (the client appends /v1/messages).

DEFAULT_MAX_TOKENS module-attribute

DEFAULT_MAX_TOKENS = 1024

max_tokens sent when neither the request nor the client sets one.

MODEL_ENV module-attribute

MODEL_ENV = 'AGENTLENS_MODEL'

Environment variable read by get_client when no spec is given.

PROVIDERS module-attribute

PROVIDERS: dict[str, _Builder] = {'openai': _openai, 'anthropic': _anthropic, 'ollama': _local('ollama', 'OLLAMA_BASE_URL', _OLLAMA_DEFAULT, None), 'vllm': _local('vllm', 'VLLM_BASE_URL', _VLLM_DEFAULT, 'VLLM_API_KEY'), 'lmstudio': _local('lmstudio', 'LMSTUDIO_BASE_URL', _LMSTUDIO_DEFAULT, None), 'openai-compat': _openai_compat}

Provider name -> builder (model, overrides) -> LLMClient. mock is handled separately (it needs no model and is not priced).

ScriptItem module-attribute

ScriptItem: TypeAlias = LLMResponse | str | ToolCall | list[ToolCall]

What a script may produce: a full response, plain text, or tool call(s).

DEFAULT_OPENAI_BASE_URL module-attribute

DEFAULT_OPENAI_BASE_URL = 'https://api.openai.com/v1'

The base_url of the official OpenAI API.

PRICING_ENV module-attribute

PRICING_ENV = 'AGENTLENS_PRICING'

Environment variable naming a YAML pricing file merged over the packaged table.

Role module-attribute

Role = Literal['system', 'user', 'assistant', 'tool']

Who authored a Message.

RetryPolicy dataclass

RetryPolicy(*, max_retries: int = 2, initial_delay: float = 0.5, max_delay: float = 8.0, max_retry_after: float = 60.0, jitter: Callable[[float], float] = equal_jitter, sleep: Callable[[float], None] = sleep, async_sleep: Callable[[float], Awaitable[None]] = sleep)

How an adapter retries failed requests.

Parameters:

Name	Type	Description	Default
max_retries	int	Retries after the first attempt (0 disables retrying).	2
initial_delay	float	Backoff before the first retry, in seconds.	0.5
max_delay	float	Upper bound of the exponential backoff, in seconds.	8.0
max_retry_after	float	Upper bound applied to a server's Retry-After, in seconds.	60.0
jitter	Callable[[float], float]	Maps the computed backoff to the actual delay; lambda d: d disables jitter (handy in tests). Not applied to Retry-After.	equal_jitter
sleep	Callable[[float], None]	Blocking sleep used by sync calls (inject a fake in tests).	sleep
async_sleep	Callable[[float], Awaitable[None]]	Sleep used by async calls.	sleep

backoff

backoff(retry_index: int) -> float

Jittered exponential backoff before retry number retry_index (0-based).

delay

delay(retry_index: int, retry_after: float | None = None) -> float

Delay before retry retry_index: the server's hint if any, else backoff.

AnthropicClient

AnthropicClient(model: str | None = None, *, base_url: str = DEFAULT_ANTHROPIC_BASE_URL, api_key: str | None = None, max_tokens: int = DEFAULT_MAX_TOKENS, timeout: float | Timeout = 60.0, max_retries: int = 2, retry_policy: RetryPolicy | None = None, headers: Mapping[str, str] | None = None, provider: str = 'anthropic', http_client: Client | None = None, async_http_client: AsyncClient | None = None)

An LLMClient for the Anthropic Messages API.

Parameters:

Name	Type	Description	Default
model	str | None	Default model (e.g. "claude-3-5-haiku-latest"), used when a request has no model. One of the two must be set.	None
base_url	str	API root without /v1 (a proxy or gateway, for instance).	DEFAULT_ANTHROPIC_BASE_URL
api_key	str | None	Sent as x-api-key. None falls back to the ANTHROPIC_API_KEY environment variable; when neither is set no key header is sent (e.g. behind a gateway that adds it). Pass "" to never send a key, even if the variable is set.	None
max_tokens	int	max_tokens used when a request has none (Anthropic requires the field).	DEFAULT_MAX_TOKENS
timeout	float | Timeout	Per-request timeout in seconds (or an httpx.Timeout).	60.0
max_retries	int	Retries on HTTP 408/429/5xx (including Anthropic's 529 "overloaded") and transport errors, with exponential backoff (Retry-After is honoured). Ignored when retry_policy is given.	2
retry_policy	RetryPolicy | None	Full control over retries/backoff/jitter (see RetryPolicy).	None
headers	Mapping[str, str] | None	Extra HTTP headers sent with every request (e.g. anthropic-beta).	None
provider	str	Value of the provider attribute that agentlens.wrap_llm records (default "anthropic").	'anthropic'
http_client	Client | None	An httpx.Client to use for sync calls (not closed by this client).	None
async_http_client	AsyncClient | None	An httpx.AsyncClient to use for async calls.	None

The API key is never logged, never part of repr and never copied into LLMResponse.raw or error messages. Use the client as a context manager (or call close) to release its connection pool.

retry_policy property

retry_policy: RetryPolicy

The retry policy in use.

url property

url: str

The Messages API endpoint.

complete

complete(request: LLMRequest) -> LLMResponse

Send request to /v1/messages and return the normalised response.

Raises:

Type	Description
LLMError	Subclasses for HTTP errors (LLMHTTPError, LLMRateLimitError), unreachable servers (LLMConnectionError, LLMTimeoutError) and unparsable answers (LLMResponseError); the base class for requests that cannot be expressed (no model, only system messages, unsupported response_format).

acomplete async

acomplete(request: LLMRequest) -> LLMResponse

Async variant of complete.

build_payload

build_payload(request: LLMRequest) -> dict[str, Any]

The JSON body sent for request (useful for debugging and tests).

close

close() -> None

Release the sync connection pool (an injected http_client is left open).

LLMClient

Bases: Protocol

A minimal, provider-neutral LLM client.

Implementations translate LLMRequest to a provider call and the provider's answer back to LLMResponse. The protocol is structural: any object with these two methods is a client, no subclassing needed.

complete

complete(request: LLMRequest) -> LLMResponse

Run request and block until the response is available.

acomplete async

acomplete(request: LLMRequest) -> LLMResponse

Async variant of complete.

LLMConfigError

Bases: LLMError, ValueError

Invalid client configuration: a bad get_client spec or pricing file.

Also a ValueError, since it is raised for bad input before any call.

LLMConnectionError

Bases: LLMError

The provider could not be reached (DNS, refused connection, broken stream...).

Raised after the retries are exhausted; the underlying httpx error is the __cause__.

LLMError

Bases: RuntimeError

Base class for failures of an LLM adapter call.

LLMHTTPError

LLMHTTPError(message: str, *, status_code: int, body: str = '')

Bases: LLMError

The provider answered with a non-success HTTP status.

Attributes:

Name	Type	Description
status_code		The HTTP status.
body		The response body (text, truncated to a few KB) for debugging.

LLMRateLimitError

LLMRateLimitError(message: str, *, status_code: int, body: str = '')

Bases: LLMHTTPError

HTTP 429: still rate limited after all retries.

LLMResponseError

Bases: LLMError

A success response that cannot be understood (not JSON, no choices, bad tool arguments).

LLMTimeoutError

Bases: LLMConnectionError

The request timed out (connect, read, write or pool timeout).

MockExhaustedError

Bases: MockLLMError

A list script ran out of responses (and cycle is off).

MockLLM

MockLLM(script: Script, *, model: str = 'mock', cycle: bool = False, default: ScriptItem | None = None)

A deterministic LLM client driven by a script.

script is one of:

• a sequence of items, returned in order, one per call. When exhausted it raises MockExhaustedError (loud failure: an agent making more calls than scripted is usually a bug), or starts over if cycle=True;
• a callable (request) -> item, for responses computed from the request;
• a mapping pattern -> item: the first pattern (in insertion order) that re.search-matches the last user message wins; otherwise default is returned, or MockNoMatchError is raised if there is no default.

With a sequence script, default (if given) also answers every call after the script is exhausted, instead of raising.

Items are an LLMResponse, plain text, a ToolCall or a list of them. Every response gets model (the request's, else this client's) and, unless the item set its own, deterministic usage: whitespace word counts of the request messages (input) and of the response (output), at cost_usd=0.

Every request is recorded in requests for assertions in tests.

calls property

calls: int

Number of requests served (or attempted) so far.

reset

reset() -> None

Forget recorded requests and restart a list script from its first item.

complete

complete(request: LLMRequest) -> LLMResponse

Return the next scripted response for request.

acomplete async

acomplete(request: LLMRequest) -> LLMResponse

Async variant of complete (no I/O, returns immediately).

MockLLMError

Bases: RuntimeError

Base class for scripting errors raised by MockLLM.

MockNoMatchError

Bases: MockLLMError

No pattern of a regex script matched and no default was given.

OpenAICompatClient

OpenAICompatClient(model: str | None = None, *, base_url: str = DEFAULT_OPENAI_BASE_URL, api_key: str | None = None, timeout: float | Timeout = 60.0, max_retries: int = 2, retry_policy: RetryPolicy | None = None, headers: Mapping[str, str] | None = None, provider: str | None = None, http_client: Client | None = None, async_http_client: AsyncClient | None = None)

An LLMClient for OpenAI-compatible /chat/completions APIs.

Parameters:

Name	Type	Description	Default
model	str | None	Default model, used when a request has no model. One of the two must be set.	None
base_url	str	API root, e.g. http://localhost:11434/v1 (Ollama), http://localhost:8000/v1 (vLLM), http://localhost:1234/v1 (LM Studio).	DEFAULT_OPENAI_BASE_URL
api_key	str | None	Sent as Authorization: Bearer .... None falls back to the OPENAI_API_KEY environment variable; when neither is set no Authorization header is sent (fine for local servers). Pass "" to never send a key, even if the variable is set.	None
timeout	float | Timeout	Per-request timeout in seconds (or an httpx.Timeout).	60.0
max_retries	int	Retries on HTTP 408/429/5xx and transport errors, with exponential backoff (Retry-After is honoured). Ignored when retry_policy is given.	2
retry_policy	RetryPolicy | None	Full control over retries/backoff/jitter (see RetryPolicy).	None
headers	Mapping[str, str] | None	Extra HTTP headers sent with every request (e.g. OpenAI-Organization).	None
provider	str | None	Value of the provider attribute that agentlens.wrap_llm records; defaults to "openai" for the official API and "openai_compat" for any other base_url.	None
http_client	Client | None	An httpx.Client to use for sync calls (not closed by this client).	None
async_http_client	AsyncClient | None	An httpx.AsyncClient to use for async calls.	None

The API key is never logged, never part of repr and never copied into LLMResponse.raw or error messages. Use the client as a context manager (or call close) to release its connection pool.

retry_policy property

retry_policy: RetryPolicy

The retry policy in use.

url property

url: str

The chat completions endpoint.

complete

complete(request: LLMRequest) -> LLMResponse

Send request to /chat/completions and return the normalised response.

Raises:

Type	Description
LLMError	Subclasses for HTTP errors (LLMHTTPError, LLMRateLimitError), unreachable servers (LLMConnectionError, LLMTimeoutError) and unparsable answers (LLMResponseError).

acomplete async

acomplete(request: LLMRequest) -> LLMResponse

Async variant of complete.

build_payload

build_payload(request: LLMRequest) -> dict[str, Any]

The JSON body sent for request (useful for debugging and tests).

close

close() -> None

Release the sync connection pool (an injected http_client is left open).

ModelPrice

Bases: SchemaModel

Price of one model, in USD per 1M tokens.

input instance-attribute

input: NonNegativeFloat

USD per 1M input (prompt) tokens.

output instance-attribute

output: NonNegativeFloat

USD per 1M output (completion) tokens.

cost

cost(input_tokens: int, output_tokens: int) -> float

USD cost of a call with these token counts.

PricedClient

PricedClient(client: LLMClient, pricing: Pricing | None = None, *, provider: str | None = None)

Wrap an LLMClient so responses carry usage.cost_usd.

Costs the provider already reported are kept; otherwise the cost is computed from pricing using the response's model (else the request's, else the client's model). Unknown models keep cost_usd=None. Attributes not defined here (model, provider, close...) are looked up on the wrapped client, so agentlens.wrap_llm still sees the provider and default model.

Parameters:

Name	Type	Description	Default
client	LLMClient	The client to wrap.	required
pricing	Pricing | None	The table to use (default: Pricing.load).	None
provider	str | None	Provider name for the free-provider check; defaults to the client's provider attribute when it is a string.	None

client instance-attribute

client = client

The wrapped client.

pricing instance-attribute

pricing = pricing if pricing is not None else Pricing.load()

The pricing table in use.

provider instance-attribute

provider = provider

Provider used for the free-provider check (None if unknown).

complete

complete(request: LLMRequest) -> LLMResponse

Call the wrapped client and fill in the response's cost.

acomplete async

acomplete(request: LLMRequest) -> LLMResponse

Async variant of complete.

Pricing

Pricing(models: Mapping[str, ModelPrice | Mapping[str, float]] | None = None, *, aliases: Mapping[str, str] | None = None, free_providers: Iterable[str] = (), updated: str | None = None)

A table of model prices with alias and prefix matching.

Parameters:

Name	Type	Description	Default
models	Mapping[str, ModelPrice | Mapping[str, float]] | None	Model id -> ModelPrice (or a {"input": .., "output": ..} mapping).	None
aliases	Mapping[str, str] | None	Model id -> the priced id it costs the same as.	None
free_providers	Iterable[str]	Providers whose models all cost 0 (local servers).	()
updated	str | None	Free-form date of the prices, for display.	None

Lookup (case-insensitive) of a model id: exact key, then alias, then the longest key that is a prefix of the id followed by -, : or @ — so dated or tagged ids (gpt-4o-mini-2024-07-18, claude-3-5-haiku-latest) use their base model's price.

from_dict classmethod

from_dict(data: Mapping[str, Any]) -> Pricing

Build a table from parsed YAML/JSON (keys models, aliases, free_providers, updated).

Raises:

Type	Description
LLMConfigError	The data does not have that shape.

from_yaml classmethod

from_yaml(text: str, *, source: str = '<string>') -> Pricing

Parse a YAML pricing table (an empty document is an empty table).

Raises:

Type	Description
LLMConfigError	Invalid YAML or an unexpected shape.

from_file classmethod

from_file(path: str | PathLike[str]) -> Pricing

Read a YAML pricing table from path.

Raises:

Type	Description
LLMConfigError	The file is missing, unreadable or invalid.

packaged classmethod

packaged() -> Pricing

The table shipped with AgentLens (approximate prices, see its updated date).

load classmethod

load(path: str | PathLike[str] | None = None) -> Pricing

The packaged table with path (else $AGENTLENS_PRICING, if set) merged over it.

merged

merged(other: Pricing) -> Pricing

A new table: this one with other's models, aliases and free providers on top.

lookup

lookup(model: str) -> ModelPrice | None

The price of model (exact, alias, then prefix match), or None if unknown.

cost

cost(model: str | None, input_tokens: int, output_tokens: int, *, provider: str | None = None) -> float | None

USD cost of a call, 0.0 for a free provider, None for an unknown model.

apply

apply(response: LLMResponse, *, model: str | None = None, provider: str | None = None) -> LLMResponse

response with usage.cost_usd filled in, when it can be.

The response's own model (the one that actually answered) wins over model. Returned unchanged when there is no usage, the cost is already set, or the model is unknown.

LLMRequest

Bases: SchemaModel

Everything needed to make one LLM call, independent of the provider.

messages also accepts a plain string, shorthand for a single user message. None values mean "provider/client default".

model class-attribute instance-attribute

model: str | None = None

None lets the client use its configured model.

response_format class-attribute instance-attribute

response_format: dict[str, JsonValue] | None = None

E.g. {"type": "json_object"} or {"type": "json_schema", "json_schema": {...}}.

last_user_message property

last_user_message: str | None

Content of the most recent user message, or None if there is none.

canonical_json

canonical_json() -> str

Stable, compact JSON encoding: sorted keys, no whitespace, unicode kept.

Two requests that are equal as models always encode to the same string, whatever the dict insertion order of tool schemas or response_format.

fingerprint

fingerprint() -> str

SHA-256 hex digest of canonical_json (the cassette request key).

LLMResponse

Bases: SchemaModel

The normalised result of one LLM call.

content class-attribute instance-attribute

content: str | None = None

Text output; None when the model only returned tool calls.

usage class-attribute instance-attribute

usage: Usage | None = None

None when the provider did not report usage.

raw class-attribute instance-attribute

raw: JsonValue = None

The provider's raw JSON payload, kept for debugging; not used for logic.

model class-attribute instance-attribute

model: str | None = None

The model that actually answered, as reported by the provider.

to_message

to_message() -> Message

This response as an assistant message, to append to the conversation.

Message

Bases: SchemaModel

One chat message.

Invariants: only assistant messages carry tool_calls; tool messages must name the call they answer in tool_call_id (and only they may); every role except assistant needs content (an assistant message may be tool calls only).

system classmethod

system(content: str) -> Message

A system prompt.

user classmethod

user(content: str) -> Message

A user turn.

assistant classmethod

assistant(content: str | None = None, tool_calls: list[ToolCall] | None = None) -> Message

An assistant turn: text, tool calls, or both.

tool classmethod

tool(content: str, tool_call_id: str) -> Message

The result of the tool call tool_call_id.

ToolCall

Bases: SchemaModel

A request from the model to call one tool.

id instance-attribute

id: str

Provider-assigned id; the matching tool message echoes it as tool_call_id.

arguments class-attribute instance-attribute

arguments: dict[str, JsonValue] = Field(default_factory=dict)

Parsed (not string-encoded) JSON arguments.

ToolSpec

Bases: SchemaModel

A tool the model may call, described by a JSON Schema for its arguments.

parameters class-attribute instance-attribute

parameters: dict[str, JsonValue] = Field(default_factory=_empty_object_schema)

JSON Schema of the arguments object.

get_client

get_client(spec: str | None = None, *, pricing: Pricing | str | PathLike[str] | None = None, **overrides: Any) -> LLMClient

Build an LLM client from "<provider>:<model>" (default: $AGENTLENS_MODEL).

Providers and the environment variables they read (overrides win):

• openai:<model> — OpenAICompatClient; OPENAI_API_KEY, OPENAI_BASE_URL (default https://api.openai.com/v1).
• anthropic:<model> — AnthropicClient; ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL.
• ollama:<model> — OLLAMA_BASE_URL (default http://localhost:11434/v1), no key.
• vllm:<model> — VLLM_BASE_URL (default http://localhost:8000/v1), optional VLLM_API_KEY.
• lmstudio:<model> — LMSTUDIO_BASE_URL (default http://localhost:1234/v1).
• openai-compat:<model> — any other compatible server: OPENAI_COMPAT_BASE_URL (required, or base_url=), optional OPENAI_COMPAT_API_KEY.
• mock / mock:<name> — MockLLM reporting model <name> (default "mock") that echoes the last user message; pass script= (and cycle=, default=) to script it. Costs 0 and needs no network.

Local servers never receive OPENAI_API_KEY. Every client except the mock is wrapped in a PricedClient, which fills usage.cost_usd from pricing: a Pricing, a YAML file merged over the packaged table, or None for Pricing.load (packaged table + $AGENTLENS_PRICING).

Raises:

Type	Description
LLMConfigError	No spec and no AGENTLENS_MODEL, an invalid spec, an unknown provider, a missing required setting or an invalid pricing file.

parse_spec

parse_spec(spec: str) -> tuple[str, str]

Split "provider:model" into a normalised (provider, model).

The provider is lower-cased and aliases resolved (openai_compat -> openai-compat, lm-studio -> lmstudio); "mock" alone means ("mock", "mock").

Raises:

Type	Description
LLMConfigError	Empty spec, missing provider: / model, or unknown provider.

count_tokens

count_tokens(text: str | None) -> int

Deterministic stand-in for a tokenizer: the number of whitespace-separated words.