Anthropic Prompt Caching API

In the previous lesson, we discussed building your own Cache to avoid calling the API entirely.

But what if you have to call the API, and you are passing the exact same 10,000-page handbook in the System Prompt every single time? Even if the user asks a different question, Claude still has to read those same 10,000 pages, and you get billed for those Input Tokens over and over.

To solve this, Anthropic introduced native Prompt Caching.

How Prompt Caching Works

Prompt Caching allows you to tell Anthropic's servers: "Keep this specific block of text in your RAM. Don't throw it away."

When the next user makes a request, if they include that exact same block of text, Anthropic's servers recognize it. Because Claude has already read it, it doesn't need to read it again.

The Benefits:

1. Cost: Cached input tokens are significantly cheaper (often 50% to 90% cheaper) than standard input tokens.
2. Speed: Because Claude doesn't have to re-read the 10,000 pages, the "Time to First Token" (TTFT) drops dramatically, making your app feel incredibly fast.

Implementing Prompt Caching

To use Prompt Caching, you add a cache_control block to the specific piece of text (or system prompt) you want Anthropic to remember.

const response = await anthropic.messages.create({
  model: 'claude-3-5-sonnet-20240620', // Check docs for supported models
  max_tokens: 500,
  system: [
    {
      type: "text",
      text: "Here is the 10,000 page handbook...",
      cache_control: { type: "ephemeral" } // THIS tells Anthropic to cache it!
    },
    {
      type: "text",
      text: "You are a helpful HR assistant."
    }
  ],
  messages: [
    { role: 'user', content: 'What is the remote work policy?' }
  ]
});

Ephemeral Lifespans

Notice the type: "ephemeral".

Prompt caches on Anthropic's servers only live for a short time (usually around 5 minutes). If no one makes an API request using that exact handbook within 5 minutes, the cache expires and is deleted.

The next time a request comes in, you will pay full price for the input tokens, and the 5-minute timer will start over.

Because of this 5-minute window, Prompt Caching is only useful for high-volume applications. If your app gets 1 request per hour, Prompt Caching will do nothing for you. If your app gets 100 requests per minute, Prompt Caching will cut your API bill in half.