Prompt Caching Cut Our AI Costs by 70%

We were bleeding money on AI API costs. Then we discovered prompt caching.

The Problem

Our AI customer service bot was handling 100K conversations/month. Each conversation:

• 4KB system prompt

• 2KB few-shot examples

• 1KB user context

• Variable user messages

Cost: $50,000/month 💸

The Insight

90% of our tokens were static:

• System prompt: Same for everyone

• Few-shot examples: Rarely change

• User context: Same within a session

We were paying to send the same tokens over and over.

Solution: Prompt Caching

Anthropic's Cache

import Anthropic from "@anthropic-ai/sdk";
const anthropic = new Anthropic();
const response = await anthropic.messages.create({
  model: "claude-3-5-sonnet-20241022",
  max_tokens: 1024,
  system: [
    {
      type: "text",
      text: systemPrompt, // 4KB of instructions
      cache_control: { type: "ephemeral" } // 👈 Cache this
    },
    {
      type: "text", 
      text: fewShotExamples, // 2KB of examples
      cache_control: { type: "ephemeral" } // 👈 Cache this too
    }
  ],
  messages: [{ role: "user", content: userMessage }]
});
// Cache hit = 90% cost reduction on cached tokens!

OpenAI's Approach

// OpenAI uses automatic caching for identical prefixes
// Just structure your prompts consistently:
const messages = [
  { role: "system", content: STATIC_SYSTEM_PROMPT },
  { role: "user", content: STATIC_CONTEXT },
  { role: "user", content: dynamicUserInput }, // Only this varies
];
// OpenAI automatically caches the static prefix

Self-Hosted with Semantic Cache

import { SemanticCache } from "ai-cache";
const cache = new SemanticCache({
  redis: redisClient,
  embedding: openai.embeddings,
  threshold: 0.95, // Similarity threshold
});
async function query(prompt: string) {
  // Check semantic cache
  const cached = await cache.get(prompt);
  if (cached) return cached;
  
  // Miss - call API
  const response = await llm.complete(prompt);
  await cache.set(prompt, response);
  return response;
}

Advanced Strategies

1. Prompt Deduplication

// Before: Redundant context in every message
messages.push({ role: "user", content: Context: ${ctx}\n\nQuestion: ${q1} });
messages.push({ role: "user", content: Context: ${ctx}\n\nQuestion: ${q2} });
// After: Context once, questions reference it
messages.push({ role: "system", content: Context: ${ctx} });
messages.push({ role: "user", content: q1 });
messages.push({ role: "user", content: q2 });

2. Response Caching

// Cache entire responses for common queries
const cacheKey = hash(prompt + model + temperature);
const cached = await redis.get(cacheKey);
if (cached) return JSON.parse(cached);

3. Tiered Models

// Use cheaper models for simple queries
const complexity = await classifier.analyze(query);
const model = complexity > 0.7 ? "gpt-4" : "gpt-3.5-turbo";

Results

| Metric | Before | After |
|--------|--------|-------|
| Monthly cost | $50,000 | $15,000 |
| Avg latency | 2.1s | 0.8s |
| Cache hit rate | 0% | 73% |

Key Takeaways

1. Audit your prompts - Find the static parts
2. Structure for caching - Put static content first
3. Monitor hit rates - Optimize based on data
4. Combine strategies - Caching + model routing + response cache

The best AI cost optimization doesn't sacrifice quality. It eliminates waste.