Prompt caching

Repeated prompt prefixes are cached automatically — they skip reprocessing, and every response reports how many tokens were served from cache.

Cura 1T is a research model, not a medical service, and not a substitute for a clinician. Benchmark scores do not establish safety for unsupervised clinical use.

How it works

When a request repeats a prompt prefix the server has recently seen — a long system prompt, a policy document, earlier conversation turns — those tokens are served from cache instead of being reprocessed. No configuration, no cache-control headers: it's automatic. The response reports what was cached:

JSON
"usage": {  "prompt_tokens": 8412,  "completion_tokens": 310,  "total_tokens": 8722,  "prompt_tokens_details": { "cached_tokens": 8192 }}

Getting the most from it

  • Put stable content first. Caching matches prefixes, so order prompts as: system prompt → reference documents → conversation history → the new user turn.
  • Keep the prefix byte-identical. Any change — a timestamp, a reordered field — invalidates the match from that point on.
  • Multi-turn conversations benefit automatically: each request re-sends history, and the unchanged turns are cache hits (see Multi-turn chat).
A clinical assistant with an 8K-token system prompt serving 1,000 requests/day serves over 8M input tokens/day from cache — with zero code changes, and every response tells you exactly how much was cached.
← JSON modeAutomatic retry →