Cold Cache and Warm Cache LLM Tests Need Separate Results

A cache-friendly benchmark can conceal the experience of a first request. Record cold and warm paths separately when evaluating a repeated prompt workflow.

Describe the repeated prefix

Identify which part of the request remains stable and which part changes. Keep the exact model, prompt format and cache configuration in the test record. A cached document lookup is a different workload from a stream of unrelated questions; neither should silently stand in for the other.

Measure both paths

Run a first request without assuming a cache hit, then repeat the intended reusable input. Inspect provider-reported cache usage where available. Record request time, usage and acceptance for each attempt. Do not infer a cache hit solely from a quick answer: short outputs and changing load can also affect duration.

Include cache costs

Record cache creation or storage charges according to the provider’s current rules. A rarely reused large prefix can cost more to keep than it saves. Estimate the number of real reuses in your application and compare that estimate with observed traffic before enabling a long retention period.

Publish a reproducible note

Report first-request and repeat-request results separately, including sample count and failures. Keep the workload description beside the chart. Gemini and Claude document their own caching mechanisms; use the documentation matching your endpoint rather than copying assumptions between providers.

Source and scope

Reference checked on 7 September 2026: Gemini context caching. This article provides an editorial workflow. It does not report a paid account test or guarantee a platform outcome.

Define cold and warm for the specific integration

Write what state exists before each request. A cold test may mean that your application has not created reusable context; a warm test may mean that the provider reports reused input under its documented mechanism. Those definitions depend on the endpoint and configuration. Do not assume that restarting your client clears a provider-side cache, or that repeating identical text guarantees a hit. Keep the observable usage evidence beside the label. When the provider gives insufficient evidence, describe the request sequence without claiming a confirmed cache state.

Hold the changing portion under control

Use a stable document or instruction prefix and a small set of representative questions. Keep the model identifier, generation settings and expected answer length comparable. If the warm request asks an easier question, a faster response may reflect the workload rather than caching. Include several questions that reuse the same material in the way your application actually does. Save a non-sensitive version of the fixture so another engineer can repeat the setup. The fixture should test the intended reusable path, not a specially constructed prompt that never appears in production.

Measure the intervals that matter

Record request start, first visible output where streaming applies, and completion. Cache behaviour can affect one interval differently from another, while output length influences total duration. Keep unsuccessful requests in the log. Report sample counts and the exact measurement points rather than one unlabeled speed number. If your application performs retrieval before the model call, measure that step separately. A model endpoint can become faster while the full user-facing workflow barely changes because another component dominates the elapsed time.

Keep creation, reuse and retention in the calculation

Consult the current provider documentation for the applicable billing categories. A reuse rate should not be applied to every token merely because part of the prompt was cached. Likewise, any creation or storage charge belongs in the complete experiment. Use the expected number of real reuses to estimate whether the mechanism makes sense for your application. If most users ask only one question about a document, a warm-heavy benchmark may describe an uncommon path. Weight the decision using observed request patterns when you have them, and label an unverified forecast as a forecast.

An example of a useful results table

Create rows for first request, immediate reuse and delayed reuse. For each row, show sample count, reported reused input, accepted outputs, first-output timing and total timing. Add a note when cache evidence is missing. You do not need to turn a small sample into a percentile chart with unwarranted precision. A few clearly described observations can reveal whether the integration works as intended. They cannot establish a universal provider ranking, especially when region, load or model configuration differs from the environment where the application will run.

Choose a deployment rule from both paths

Set an acceptable first-request experience as well as a target for repeat work. If the cold path misses the application’s deadline, warm performance alone may not justify deployment. Consider whether the application can prepare reusable context in advance under the provider’s supported workflow and your privacy requirements. Keep a fallback that does not assume a hit. Recheck the experiment when the prefix, model or traffic pattern changes. A cache optimization remains useful only while the application actually produces the conditions under which you measured it.