Claude API Prompt Caching: A Guide to Lowering LLM Costs

What is Claude API Prompt Caching?

In any application using a Large Language Model (LLM) like Anthropic's Claude, a significant portion of the cost comes from processing the input prompt. This prompt often includes a long system message, user history, or retrieved documents that remain the same across many user interactions. Sending this identical data repeatedly is inefficient and expensive.

Claude API prompt caching is a feature offered by Anthropic that solves this problem. It allows the API to store and reuse the computed state of a large, static portion of your prompt. When you make a subsequent API call with the same cached prompt, you only send the new, variable part of the input. The API then uses the stored state, processes only the new tokens, and generates a response. This results in dramatically lower costs and faster response times.

At JRV Systems, we see this as the single most impactful cost-saving feature for any production-grade LLM application today, especially for systems with structured, repetitive tasks.

How Caching Works: TTL and Cache Breakpoints

Understanding the mechanics of prompt caching is key to using it effectively. The system is straightforward but has specific rules.

First, you enable caching by including the "cache_control": {"type": "ephemeral"} parameter in your API request. When Claude receives this, it caches the prompt (specifically, everything except the last user message). The cache entry is assigned a Time-To-Live (TTL) of 24 hours, meaning it will be available for reuse for one day.

To use the cached prompt, your next API call must meet two conditions:

• Identical Prefix: The system prompt and all messages, except for the final user message, must be exactly the same as the original cached request. The order and content must match character for character.
• Model and Version: The request must use the same model version (e.g., claude-3-5-sonnet-20240620).

Any change to the static part of the prompt—even a single space—will cause a "cache miss." The API will then re-process the entire prompt and create a new cache entry. This is known as a cache breakpoint. Effectively managing these breakpoints is crucial for maintaining a high cache hit rate.

A Real-World Example: A Malaysian SaaS with 50k Support Tickets

Let's consider a practical scenario. Imagine a Malaysian e-commerce platform that uses a Claude-powered chatbot to handle 50,000 customer support queries per month. For each query, the chatbot needs context.

The prompt for every interaction looks like this:

• System Prompt: Detailed instructions on how to behave, the company's return policy, and product information. (6,000 tokens)
• User Chat History: The last few messages from the current conversation. (2,000 tokens)
• New User Question: The actual new query from the customer. (e.g., "Where is my order #MY12345?") (50 tokens)

Without caching, the application sends all 8,050 tokens of this prompt to the Claude API every single time. The model then generates a response, let's say an average of 400 tokens.

With Claude API prompt caching, the 8,000-token block (System Prompt + Chat History) is cached on the first request. For all subsequent messages in that same conversation, the application only needs to send the new 50-token user question. The API retrieves the 8,000-token context from its cache, processes the new 50 tokens, and generates the response.

Calculating the Cost Savings

The financial impact is substantial. Let's use the pricing for Claude 3.5 Sonnet (as of mid-2024):

• Standard Input: $3.00 per million tokens
• Standard Output: $15.00 per million tokens
• Cached Input (Cache Hit): $1.50 per million tokens (50% cheaper)

Let's calculate the cost for a single support query, assuming a cache hit.

Cost Without Caching:

• Input cost: (8,050 tokens / 1,000,000) * $3.00 = $0.02415
• Output cost: (400 tokens / 1,000,000) * $15.00 = $0.00600
• Total per query: $0.03015

Cost With Caching:

• Cached prompt cost: (8,000 tokens / 1,000,000) * $1.50 = $0.01200
• New input cost: (50 tokens / 1,000,000) * $3.00 = $0.00015
• Output cost: (400 tokens / 1,000,000) * $15.00 = $0.00600
• Total per query: $0.01815

This is a 40% cost reduction per query. Over 50,000 queries per month, assuming a high cache hit rate for conversations, the savings become significant:

• Monthly Cost (No Cache): 50,000 * $0.03015 = $1,507.50
• Monthly Cost (With Cache): 50,000 * $0.01815 = $907.50

That's a saving of $600 USD (around RM2,800) per month. The latency also improves because the model has much less data to process on each turn.

When Caching is Most Effective

Prompt caching isn't a universal solution, but it excels in specific use cases that are common in SaaS products we build for our clients in Malaysia.

• Chatbots & Conversational Agents: Where a long conversation history or system prompt is maintained.
• RAG (Retrieval-Augmented Generation): When querying documents, the retrieved text chunks can be cached as part of the prompt, and only the user's question changes.
• Code Generation Tools: The surrounding code files and project context can be cached.
• Structured Data Analysis: A tool that repeatedly analyzes data against a fixed set of instructions or a schema.

Essentially, any application where a large part of the prompt is static and a small part is dynamic is a prime candidate for Claude API prompt caching. It's a simple, server-side change that requires no complex engineering but delivers immediate and measurable returns on your AI operational expenditure.