How to Test LLM Applications Without Calling the Model

The suite went red on a Tuesday. Nothing had changed since Friday. Same commit, same lockfile, same fixtures. The failing assertion compared a model's answer against a string somebody had pasted into the test six weeks earlier, and the model had decided to open with "Sure" instead of "Certainly".

Most of what gets written about how to test LLM applications is really about evaluating model quality: rubrics, judges, golden datasets, scores that drift. That work matters and it belongs in its own pipeline with its own budget. This article is about the other half. The system around the model is ordinary software. Retry wrappers, schema parsers, tool dispatchers, state machines, caches, queue consumers and database writes. All of it can be tested the way software has always been tested, deterministically, on every commit, in under a minute. What that requires is deciding exactly where determinism stops and defending that line.

The seam is the model client

Nothing downstream is testable until the model can be replaced. That means the model call sits behind an interface you own rather than a vendor SDK type threaded through twelve files. One seam, one place to swap.

Pydantic AI ships this as a first-class idea and it is worth copying even if you write Go or TypeScript. TestModel calls every tool the agent exposes and generates data that satisfies each tool's JSON schema using plain procedural Python, no model involved. FunctionModel hands you the response function so you can script exactly what comes back. Agent.override() swaps model, dependencies or toolsets inside application code without touching call sites. Then there is the piece teams forget: setting models.ALLOW_MODEL_REQUESTS = False globally, so a stray real call fails loudly instead of quietly billing you from inside a unit test. The same shape appears in .NET agent stacks built on the IChatClient abstraction, where an application constructs an agent with a real provider in production and a scripted client under test.

Two rules keep the stub honest. Make it a scripted transcript, not a clever fake. The moment your fake grows branching logic, you are testing the fake. And assert on what the stub received, not only on what it returned. Prompt assembly, context truncation, tool schema serialisation, message ordering, system prompt composition: these are pure functions with real bugs in them, and a recording stub turns every one of them into a fast, exact test. In review, most of the defects I find in agent code live in the request, not the response.

A cassette is a claim about the world with an expiry date. Treat it like a TLS certificate.

Cassettes with expiry dates

Above the stub sits replay. VCR.py, now at 8.3.0, and its pytest-recording wrapper intercept HTTP at the transport layer and write request and response pairs to a cassette file. Run once against the live API, commit the YAML, and every subsequent run replays bytes. The test then fails for exactly one reason: the request your code produced no longer matches the request that was recorded. That is a genuinely useful signal, and it is cheap.

Cassettes rot in three ways and only one of them is obvious. The model gets retired: Anthropic retired claude-3-7-sonnet on 19 February 2026 and claude-opus-4-1 on 5 August 2026, giving at least 60 days of notice each time. A cassette pinned to a retired model replays green forever while the live path returns an error. The request shape drifts: temperature, top_p and top_k are deprecated on Claude Opus 4.7 and later, and setting them to a non-default value returns a 400. Every cassette recorded under the old habit of pinning temperature to zero for reproducibility now encodes a request the API rejects. Replay is green. Production is a 400. Third, the provider adds response fields your parser never sees because the cassette predates them.

So treat cassettes as claims with a shelf life. Stamp each one with the model id, the API version and the record date in its metadata. Fail the suite when a cassette is older than your chosen window or names a model outside the current allow-list. Scrub authorization headers at record time, not in review. Re-record on a schedule rather than when something breaks, because re-recording under deadline pressure is how a broken request gets frozen into a passing test.

A cassette is a claim about the world with an expiry date. Treat it like a TLS certificate.

Assert properties, not strings

Exact-match assertions on generated text are a promise the system cannot keep. Sampling aside, providers reserve the right to change weights, tokenizers, safety filters and serving infrastructure without telling you, and even at temperature zero the batch you land in is shared with strangers. What survives all of that is properties.

Useful invariants are more specific than people expect. Every tool call the planner emits validates against that tool's input schema. The plan graph is acyclic. Line items sum to the stated total. No identifier appears in the answer that was absent from the retrieved context. Two calls with the same idempotency key produce one row, not two. A conversation that hits the token ceiling still leaves the state machine in a legal state. Hypothesis and its equivalents generate thousands of inputs against these and shrink any failure to its smallest reproduction, which is worth more than a hand-written case that happened to occur to somebody.

Metamorphic relations cover the rest. Paraphrase the input and the extracted fields should be identical even though the prose is not. Reorder retrieved chunks and the set of cited document ids should not change. Add a relevant document and grounding should not get worse. These are assertions about relationships between runs rather than about any single output, which is the only kind of assertion a sampled component can honour.

Structured outputs have quietly moved the goalposts here. When a provider compiles your JSON Schema into a grammar and masks invalid tokens during decoding, shape conformance stops being your problem on the happy path. So delete the test that asserts the JSON parses and write tests for the conditions where the guarantee lapses instead: generation truncated at the token limit, and an outright refusal. Those are the branches that ship broken.

Contract tests where the model touches your code

The riskiest interface in an agent system is the one where a probabilistic component is allowed to invoke your functions. That interface has a schema, which makes it testable in the classical consumer and provider sense.

The Model Context Protocol has been tightening exactly this. The 2026-07-28 specification moved tool inputSchema and outputSchema to full JSON Schema 2020-12 with composition, conditionals and references (SEP-2106), added a formal feature lifecycle of Active, Deprecated and Removed with at least twelve months between deprecation and earliest removal, and deprecated Roots, Sampling and Logging in that same release. It also introduced a rule worth stealing: a Standards Track proposal cannot reach Final until a matching scenario lands in the conformance suite, which is the suite official SDKs are now scored against.

The practical contract test makes the schema file the single shared artifact. On the provider side, property-test that every handler accepts everything the schema admits and rejects everything it does not, including the awkward middle where a field is present but null. On the consumer side, generate or validate the stub used in your unit tests from that same file, so a schema change breaks the fast suite rather than production. Snapshot the schema and diff it in CI. A field renamed from customer_id to customerId is invisible to every prose-level assertion and fatal at runtime.

One more asset belongs under version control with a test around it: the tool description text. Changing a description changes which tool the model reaches for, and no type system anywhere will notice. Treat descriptions as behavioural code, review them as such, and pin the ones your routing tests depend on.

The failure paths are most of the product

What pages people at 03:00 is almost never a mediocre answer. It is a partial one.

The error surface is documented and finite, which means it is enumerable in a stub. The Claude API returns 429 rate_limit_error, 500 api_error, 504 timeout_error and 529 overloaded_error, and the official SDKs already retry transient failures twice by default with exponential backoff, honouring retry-after when it is present. That default matters more than it looks. On streaming responses an error can arrive after the API has already returned 200, so mid-stream failures bypass the normal error path entirely and land in whatever your accumulator does with a truncated event sequence.

Give the fault-injecting stub one switch per failure and write a test for each. Timeout after partial tokens. Refusal. JSON truncated at the token ceiling. A tool call whose arguments validate against the schema but reference a record that does not exist. The same tool called twice in one turn. Plain text returned where a tool call was required. A 429 with retry-after and a 429 without. A 529 arriving on every attempt until the circuit breaker opens.

A pattern I have run into more than once on factory-floor systems: an incident is filed against the model and the model is innocent. The shape repeats. A 504 arrives after the vendor SDK has already retried twice, the application's own retry wrapper fires on top of that, and a single operator action lands in the database twice. Nobody notices until a downstream count disagrees with a physical count on the line. The fix is an idempotency key and a test that calls the stub three times and asserts the table holds one row. An ordinary distributed systems bug wearing an AI costume, and it will not appear in any eval you ever run.

If your fake can only succeed, your suite only covers the day nothing goes wrong.

How to test LLM applications without a five-figure CI bill

Tier the suite by what each layer costs and how often it can honestly run. Stubbed unit tests, no network, on every commit, with real model requests globally disabled. Schema contract tests, also offline, on every commit. Cassette replay on every pull request. A small live smoke suite against the pinned production model, nightly. Evals in a separate pipeline with a separate budget and their own dashboard, because mixing a statistical score into a merge gate teaches everyone to press rerun.

Caching does more work than model choice. Promptfoo keeps a disk cache on by default, keyed on provider identifier, prompt content, provider configuration and context variables, and it deliberately does not cache errors so retries stay real. Re-running an unchanged suite then costs nothing. Downgrading the model for pull request runs is the second lever, though it changes what you are measuring, so keep the nightly run on the model you actually ship.

Timezone arbitrage is real and almost always accidental. DeepSeek bills peak rates from 01:00 to 04:00 and 06:00 to 10:00 UTC on weekdays, with off-peak at half. In Shenzhen that peak window is 09:00 to 12:00 and 14:00 to 18:00 local, which is the working day here almost to the minute. A nightly live suite firing at 02:00 local time bills at 18:00 UTC the day before, sitting in the cheap half, and nobody on the team decided that. Move the same job to 10:00 so results are waiting at standup and the spend doubles. I have sat through a meeting about model costs that turned out to be a meeting about a cron expression.

Do not gate merges on a live model call. Flaky gates get rerun, reruns cost more than the original run, and a check everybody reruns has stopped being a check.

The strongest objection to all of this

Here is the argument against, at full strength. Everything above tests plumbing. The stub returns whatever you told it to return, so a green suite proves only that your fake agrees with itself. The failures that actually hurt are behavioural: the model picks the wrong tool, invents a plausible identifier, follows an injected instruction in a retrieved document, or quietly degrades when a provider updates weights. Stubs are blind to all of it. Worse, a wall of green makes teams feel covered when they are not, which is a more dangerous state than knowing you have no tests.

That objection is correct, and it describes the most common failure I see in code review. Four hundred stubbed tests, no eval set, and a regression that shipped because nobody was measuring the thing that regressed.

It is also not an argument for skipping the deterministic layer, because the two catch disjoint fault classes and the incident logs are lopsided. Truncated JSON, double-writes from stacked retries, schema drift after a tool rename, a timeout that leaves a workflow half-applied, a prompt template that silently dropped a variable: these are plumbing bugs, they are the majority of real outages in LLM-backed systems, and they are exactly what stubs catch. Behavioural regressions are real too. They need evals, and evals are a different instrument.

There is also a dependency running one way. A live eval that fails is only informative if you already know the harness is sound. Without a fast deterministic suite underneath, every eval failure starts with an hour of arguing about whether the model got worse or somebody broke the retriever. If you want to know whether your deterministic tests have teeth, mutation testing answers that question honestly and coverage percentages do not. Deterministic tests gate merges. Evals gate releases. Budget both.

Determinism you can buy

In September 2025 Thinking Machines Lab published measurements that changed how this problem should be framed. A thousand completions at temperature zero from Qwen3-235B-A22B-Instruct-2507 produced 80 unique results, with the first divergence appearing at token 103. The cause was not floating point luck. It was batch-size dependence in reduction kernels, meaning your output depends on who else was in the batch with you. With batch-invariant kernels, all 1000 completions came back identical. The cost on Qwen3-8B across 1000 sequences was 26 seconds for standard vLLM against 42 seconds with an improved batch-invariant attention kernel, and 55 seconds before that optimisation.

vLLM ships it now behind a single environment variable, VLLM_BATCH_INVARIANT=1, set before serving or before import for offline inference.

Which leaves an odd split. If you self-host, bit-exact reproducibility is now purchasable for roughly 1.6 times the latency, and your integration tests can go back to asserting on exact bytes like any other component. If you call a hosted API, you cannot buy it at any price, because the batch you land in is not yours. The teams with the most control over their own hardware need the least test cleverness. Everyone else writes property tests because reproducibility is not on their price list. And the first hosted vendor to sell a deterministic tier will turn half of this discipline into an artifact of a temporary market condition, which is worth remembering before anyone builds a career around cassette hygiene.

Tools referenced

vLLM, reviewed here: vLLM review.

Ollama, reviewed here: Ollama review.

llama.cpp, reviewed here: llama.cpp review.

LangGraph, reviewed here: LangGraph review.

OpenAI Agents SDK, reviewed here: OpenAI Agents SDK review.

Temporal, reviewed here: Temporal review.

Sources

Pydantic AI, Unit testing with TestModel and FunctionModel: https://pydantic.dev/docs/ai/guides/testing/

Claude API errors: 429, 500, 504, 529 and SDK retry behaviour: https://platform.claude.com/docs/en/api/errors

Claude model deprecations and retirement dates: https://platform.claude.com/docs/en/about-claude/model-deprecations

MCP 2026-07-28 specification release candidate: https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/

vLLM, Batch Invariance feature documentation: https://docs.vllm.ai/en/latest/features/batch_invariance/

Thinking Machines Lab, Defeating Nondeterminism in LLM Inference: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

Promptfoo caching configuration: https://www.promptfoo.dev/docs/configuration/caching/

DeepSeek API models and pricing, including off-peak windows: https://api-docs.deepseek.com/quick_start/pricing