Prompt Caching for Text-to-SQL: 80% Cost Reduction Math

If you're running text-to-SQL workloads at scale—whether embedded in dashboards, powering self-serve BI, or automating query generation—your LLM costs are probably higher than they need to be. Every time a user asks a question about their data, you're sending the same database schema, table definitions, and system instructions to Claude, GPT-4, or another model. You're paying full price for that context. Every. Single. Time.

Prompt caching flips that equation. By caching the static portions of your prompts—the schema, the instructions, the few-shot examples—you pay 90% less for cached tokens on reads and cut latency by 75%. For text-to-SQL workloads specifically, the math is brutal in your favor: an 80% cost reduction is not aspirational. It's achievable with straightforward engineering.

This article walks through the mechanics, the math, and the patterns that make it work.

What Is Prompt Caching and Why It Matters for Text-to-SQL

Prompt caching is a feature offered by major LLM providers—Anthropic's Claude, OpenAI, and Amazon Bedrock—that stores portions of your prompt in the model's KV cache on the provider's infrastructure. Once cached, subsequent requests that use the same cached tokens are charged at a fraction of the normal rate.

Here's the practical implication: if your prompt contains a 5,000-token database schema and 2,000 tokens of system instructions, and 100 users query that same database in a day, you're paying full price for those 7,000 tokens 100 times. With caching, you pay full price once, then 10% of that price for the next 99 reads.

For text-to-SQL specifically, this is a game-changer because:

• Database schemas are static or change infrequently. Your table definitions, column names, data types, and relationships don't shift every second. They're the perfect candidate for caching.
• User queries are repetitive. Multiple users ask similar questions. "What's revenue this month?" "Show me customer churn." "Break down sales by region." The schema context is identical; only the user's natural language question changes.
• The cost structure is extreme. Text-to-SQL prompts are heavy. A typical prompt includes the full schema, sample data, instructions on how to handle ambiguity, and examples of well-formed queries. That's 3,000–10,000 tokens before you even ask the user's question. Caching that context is where the leverage lives.

The math: if your average text-to-SQL prompt is 8,000 tokens (schema + instructions) plus 500 tokens (user query), and you run 1,000 queries per day against the same database:

• Without caching: 1,000 × 8,500 = 8.5M tokens/day
• With caching: 8,000 × 1 (full price) + (8,000 × 0.1) × 999 (cached reads) + 500 × 1,000 (user query) = 8,000 + 792,000 + 500,000 = 1.3M tokens/day
• Savings: (8.5M - 1.3M) / 8.5M = 84% reduction

That's not theoretical. That's what the providers deliver, and it's what you see in production systems.

Understanding the Pricing Mechanics Behind the 80% Math

The 80% figure isn't magic—it's a direct consequence of how LLM providers price cached tokens. Let's break down the actual numbers.

Anthropic's Pricing Model

Anthropic's prompt caching infrastructure offers a 90% discount on cached token reads. Here's what that means in practice:

• Input tokens (non-cached): $3 per 1M tokens for Claude 3.5 Sonnet
• Cached input tokens (reads): $0.30 per 1M tokens (90% off)
• Cache write cost: Same as non-cached input tokens; you pay full price when the cache is first populated

For a 10,000-token schema cached and read 100 times:

• First request (cache write): 10,000 × $3/1M = $0.03
• Next 99 requests (cache reads): 99 × 10,000 × $0.30/1M = $0.0297
• Total cost: $0.0597
• Cost per query: $0.0597 / 100 = $0.000597 per query

Without caching, the same 100 queries would cost: 100 × 10,000 × $3/1M = $3.00. The cached version costs $0.0597—a 98% reduction for that specific prompt segment.

OpenAI's Approach

OpenAI's prompt caching works differently but delivers similar savings. OpenAI caches prompts over 1024 tokens and charges 50% of the input token price for cached reads:

• Input tokens (non-cached): $5 per 1M tokens for GPT-4o
• Cached input tokens (reads): $2.50 per 1M tokens (50% off)

The discount is lower than Anthropic's, but the principle is identical: repeated context is cheaper on the second and subsequent uses.

Amazon Bedrock

Amazon Bedrock's prompt caching for Claude models mirrors Anthropic's 90% discount structure, since Bedrock runs Anthropic's models. Cache reads cost 10% of normal input token rates, making it attractive for workloads with high repetition.

Why the 80% Figure Is Conservative

The 80% cost reduction in the title is actually conservative. Here's why:

1. Real-world cache hit rates exceed 90%. In production text-to-SQL systems, the same database schema is queried hundreds or thousands of times per day. After the first request, nearly every subsequent query hits the cache.
2. The cached portion is the largest part of the prompt. Your schema and instructions are 80–90% of the total tokens. The user's question is 10–20%. That means the high-discount portion dominates the savings.
3. Latency improvements compound the value. Cached reads show 75% latency reduction, meaning queries return faster, which improves user experience and reduces infrastructure load.

In real systems running embedded analytics or self-serve BI on top of D23's managed Apache Superset, teams report 80–85% cost reductions once caching is properly implemented. Some hit 90% with highly repetitive workloads.

The Anatomy of a Text-to-SQL Prompt and What Gets Cached

To implement caching effectively, you need to understand what parts of your prompt are static (cacheable) and what parts change with every request.

Static Components (Cache These)

Database schema: Your entire schema definition—tables, columns, data types, primary/foreign keys, indexes. This is typically 2,000–8,000 tokens depending on database size.

TABLE: customers
  - id (INTEGER, PRIMARY KEY)
  - name (VARCHAR)
  - email (VARCHAR)
  - signup_date (DATE)
  - country (VARCHAR)

TABLE: orders
  - id (INTEGER, PRIMARY KEY)
  - customer_id (INTEGER, FOREIGN KEY -> customers.id)
  - amount (DECIMAL)
  - order_date (DATE)
  - status (VARCHAR: 'pending', 'completed', 'cancelled')

System instructions: How the model should approach query generation, handle ambiguity, optimize for performance, handle NULL values, etc. Typically 1,000–2,000 tokens.

Few-shot examples: Sample natural language questions paired with correct SQL queries. These teach the model your style and constraints. Typically 1,500–3,000 tokens.

Data dictionary or documentation: Descriptions of what each table and column represents, common joins, business logic. Typically 500–2,000 tokens.

Constraints and rules: "Never use SELECT *", "Always filter by created_date for performance", "Use this specific schema for PII data." Typically 300–800 tokens.

Total static context: 5,300–15,800 tokens depending on database complexity.

Dynamic Components (Don't Cache)

User query: The natural language question. "What's my revenue by product category this month?" Typically 50–300 tokens.

Runtime context: Current date, user permissions, filters applied by the application, session state. Typically 100–500 tokens.

Recent data or context: If you include recent rows for context, those change. Typically 0–1,000 tokens depending on your approach.

Total dynamic context: 150–1,800 tokens per request.

The Prompt Structure

A well-structured text-to-SQL prompt for caching looks like this:

[CACHE_CONTROL: static_schema]

You are a SQL expert. Generate SQL queries based on natural language questions.

[Database Schema]
[Full schema definition - 5,000+ tokens]

[Instructions]
[How to handle joins, NULL values, performance - 1,500 tokens]

[Examples]
Q: What's our top 10 customers by revenue?
A: SELECT customer_id, SUM(amount) as total_revenue FROM orders WHERE status = 'completed' GROUP BY customer_id ORDER BY total_revenue DESC LIMIT 10;

[More examples - 2,000 tokens]

[END_STATIC_CACHE]

[DYNAMIC_CONTEXT]
Current date: 2025-01-15
User timezone: UTC
User permissions: can access customers, orders, products tables

[USER_QUESTION]
What was our average order value in January?
[/USER_QUESTION]

The cache control markers tell the LLM provider what to cache. Everything between the markers is cached on the first request; on subsequent requests, only the dynamic context and user question are sent fresh.

Implementation Patterns: How to Actually Cache in Production

Caching isn't automatic—you need to structure your requests correctly. Here's how it works across providers.

Anthropic Claude Implementation

Anthropics's official documentation on prompt caching uses a cache_control parameter in the API request. Here's a Python example:

import anthropic
 
client = anthropic.Anthropic(api_key="your-api-key")
 
schema = """TABLE: customers...[5000 tokens]..."""
instructions = """You are a SQL expert...[1500 tokens]..."""
examples = """Q: What's revenue?...[2000 tokens]..."""
 
user_query = "What's our average order value this month?"
 
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": schema + instructions + examples,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {
            "role": "user",
            "content": user_query
        }
    ]
)
 
print(response.content[0].text)
print(f"Cache creation tokens: {response.usage.cache_creation_input_tokens}")
print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")
print(f"Input tokens: {response.usage.input_tokens}")

On the first request, cache_creation_input_tokens will be high (the schema, instructions, and examples). On subsequent requests with the same system prompt, cache_read_input_tokens will be high, and the cost will drop to 10% of normal.

OpenAI Implementation

OpenAI's caching works similarly but requires the cached content to be over 1024 tokens:

import openai
 
client = openai.OpenAI(api_key="your-api-key")
 
schema = """TABLE: customers...[5000 tokens]..."""
instructions = """You are a SQL expert...[1500 tokens]..."""
examples = """Q: What's revenue?...[2000 tokens]..."""
 
user_query = "What's our average order value this month?"
 
response = client.chat.completions.create(
    model="gpt-4o",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": schema + instructions + examples,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {
            "role": "user",
            "content": user_query
        }
    ]
)
 
print(response.choices[0].message.content)
print(f"Cache creation tokens: {response.usage.cache_creation_input_tokens}")
print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")

Amazon Bedrock Implementation

Amazon Bedrock's prompt caching is configured via the request body:

import boto3
 
client = boto3.client('bedrock-runtime')
 
schema = """TABLE: customers...[5000 tokens]..."""
instructions = """You are a SQL expert...[1500 tokens]..."""
examples = """Q: What's revenue?...[2000 tokens]..."""
 
user_query = "What's our average order value this month?"
 
response = client.invoke_model(
    modelId='anthropic.claude-3-5-sonnet-20241022-v2:0',
    body={
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": schema + instructions + examples,
                        "cache_control": {"type": "ephemeral"}
                    },
                    {
                        "type": "text",
                        "text": user_query
                    }
                ]
            }
        ],
        "max_tokens": 1024
    }
)
 
output = json.loads(response['body'].read())
print(output['content'][0]['text'])
print(f"Cache creation tokens: {output['usage'].get('cache_creation_input_tokens', 0)}")
print(f"Cache read tokens: {output['usage'].get('cache_read_input_tokens', 0)}")

Real-World Cost Scenarios: The Math in Action

Let's walk through three realistic scenarios and calculate actual cost savings.

Scenario 1: Mid-Market SaaS with Embedded Analytics

Setup:

• 500 active users
• 2,000 queries per day (4 queries per user on average)
• Database with 50 tables, ~12,000 tokens of schema
• System instructions and examples: ~3,500 tokens
• Average user query: 200 tokens
• Using Claude 3.5 Sonnet at $3/1M input tokens

Without caching:

• Tokens per query: 12,000 + 3,500 + 200 = 15,700
• Daily tokens: 2,000 × 15,700 = 31.4M tokens
• Daily cost: 31.4M × ($3/1M) = $94.20
• Monthly cost: $94.20 × 30 = $2,826

With caching (90% cache hit rate):

• First query of the day: 15,700 tokens at full price = 15,700 × ($3/1M) = $0.047
• Remaining 1,999 queries: (12,000 + 3,500) × 0.1 + 200 = 1,750 tokens at cached rate per query
• Cost for cached queries: 1,999 × 1,750 × ($3/1M) = $0.00525 × 1,999 = $10.49
• Daily cost: $0.047 + $10.49 = $10.54
• Monthly cost: $10.54 × 30 = $316.20

Savings: $2,826 - $316.20 = $2,509.80 per month (89% reduction)

Scenario 2: Enterprise Analytics Platform

Setup:

• 5,000 active users
• 50,000 queries per day (10 queries per user)
• Multiple databases; average 20,000 tokens of schema per database (5 databases)
• System instructions and examples: ~5,000 tokens
• Average user query: 300 tokens
• Using Claude 3.5 Sonnet

Without caching:

• Tokens per query: 20,000 + 5,000 + 300 = 25,300
• Daily tokens: 50,000 × 25,300 = 1.265B tokens
• Daily cost: 1.265B × ($3/1M) = $3,795
• Monthly cost: $3,795 × 30 = $113,850

With caching (92% cache hit rate):

• First query per database per day: 5 × 25,300 = 126,500 tokens at full price = $0.38
• Remaining 49,995 queries: (20,000 + 5,000) × 0.1 + 300 = 2,800 tokens per query
• Cost for cached queries: 49,995 × 2,800 × ($3/1M) = $419.96
• Daily cost: $0.38 + $419.96 = $420.34
• Monthly cost: $420.34 × 30 = $12,610.20

Savings: $113,850 - $12,610.20 = $101,239.80 per month (89% reduction)

Scenario 3: Private Equity Portfolio Analytics

Setup:

• 15 portfolio companies
• 100 users across all companies
• 5,000 queries per day (50 per user)
• Standardized schema across all companies: 8,000 tokens
• Shared instructions and examples: 2,500 tokens
• Average query: 250 tokens
• Using OpenAI GPT-4o at $5/1M input tokens (higher cost model)

Without caching:

• Tokens per query: 8,000 + 2,500 + 250 = 10,750
• Daily tokens: 5,000 × 10,750 = 53.75M tokens
• Daily cost: 53.75M × ($5/1M) = $268.75
• Monthly cost: $268.75 × 30 = $8,062.50

With caching (95% cache hit rate, OpenAI 50% discount):

• First query: 10,750 tokens at full price = 10,750 × ($5/1M) = $0.054
• Remaining 4,999 queries: (8,000 + 2,500) × 0.5 + 250 = 5,500 tokens per query
• Cost for cached queries: 4,999 × 5,500 × ($5/1M) = $137.47
• Daily cost: $0.054 + $137.47 = $137.52
• Monthly cost: $137.52 × 30 = $4,125.60

Savings: $8,062.50 - $4,125.60 = $3,936.90 per month (51% reduction with OpenAI)

Note: OpenAI's 50% discount is lower than Anthropic's 90%, so savings are more modest. For PE firms, switching to Anthropic or Bedrock could push savings to 80%+.

Advanced Caching Strategies: Going Beyond the Basics

Once you understand the fundamentals, there are sophisticated patterns that amplify savings.

Multi-Tier Caching

Prompt caching infrastructure for LLM cost and latency reduction describes a multi-tier approach: cache the schema once, then cache schema + instructions + examples as a second tier, then cache schema + instructions + examples + recent context as a third tier.

This is useful when:

• Your schema changes monthly but instructions change weekly
• You want to cache "hot" examples separately from "cold" documentation
• Different user cohorts need different cached contexts

Implementation: Use separate cache_control markers for each tier, and structure your prompts to reuse lower tiers.

Schema Versioning and Cache Invalidation

Your database schema will change. When it does, your cache becomes stale. Handle this by:

1. Versioning your schema: Include a schema version number in the cached prompt. When the schema changes, increment the version.
2. Automatic cache invalidation: When you detect a schema change, clear the cache and start fresh.
3. Gradual rollout: Cache both old and new schemas during a transition period, then deprecate the old version.

Example:

schema_version = "2025-01-15-v2"
schema_prompt = f"""SCHEMA VERSION: {schema_version}
 
TABLE: customers...[schema definition]
"""
 
# On schema change, increment version and new cache is created
if detect_schema_change():
    schema_version = "2025-01-20-v3"

Dynamic Few-Shot Example Selection

Instead of caching all examples, cache a base set of high-value examples, then add user-specific or context-specific examples dynamically. This keeps the cached portion large (high savings) while allowing personalization.

base_examples = """[cached examples - 2000 tokens]"""
dynamic_examples = f"""[user-specific examples - 300 tokens]"""
 
# Cache the base, send dynamic fresh
response = client.messages.create(
    system=[
        {"type": "text", "text": schema + instructions + base_examples, "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": dynamic_examples}
    ],
    messages=[{"role": "user", "content": user_query}]
)

Cross-Database Caching

If you run queries across multiple databases with similar schemas, you can cache a generic schema definition and then add database-specific details dynamically. This amortizes the cache across all databases.

generic_schema = """[Common schema patterns - 5000 tokens, cached]"""
db_specific = """[Database X specifics - 1000 tokens, dynamic]"""
 
response = client.messages.create(
    system=[
        {"type": "text", "text": generic_schema, "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": db_specific}
    ],
    messages=[{"role": "user", "content": user_query}]
)

Integration with Managed Analytics Platforms

If you're building text-to-SQL into a self-serve BI or embedded analytics product, prompt caching integrates seamlessly with platforms like D23's managed Apache Superset. Here's why it matters:

D23 provides managed Superset hosting with API-first architecture, which means:

1. Superset dashboards can trigger text-to-SQL queries via the REST API, and those queries can be cached.
2. Embedded analytics in your product can use cached prompts to generate queries on behalf of end users.
3. Self-serve BI interfaces can leverage caching to reduce costs while scaling query generation.
4. Data consulting services can standardize cached schemas across customer implementations.

When you integrate caching with D23's platform, you're not just saving on LLM costs—you're improving dashboard load times (cached reads are faster) and reducing infrastructure load on your Superset instance.

Monitoring, Metrics, and Cost Tracking

Once caching is live, you need visibility into what's working.

Key Metrics to Track

Cache hit rate: Percentage of requests that hit the cache.

• Target: 85%+ for stable workloads
• Formula: cache_read_input_tokens / (cache_read_input_tokens + input_tokens)

Cost per query: Total LLM cost divided by number of queries.

• Without caching: Typically $0.05–$0.15 per query
• With caching: Typically $0.005–$0.015 per query

Cache efficiency ratio: Tokens saved via caching divided by total tokens.

• Formula: cache_read_input_tokens / total_input_tokens
• Target: 60%+ of total tokens should be cached reads

Latency improvement: Time to first token (TTFT) and total latency with vs. without caching.

• Cached reads: 75% faster on average

Logging and Observability

Log every request with:

import logging
import json
 
logger = logging.getLogger(__name__)
 
def log_query_metrics(response, user_id, query_text):
    metrics = {
        "timestamp": datetime.now().isoformat(),
        "user_id": user_id,
        "query_text": query_text,
        "cache_creation_tokens": response.usage.cache_creation_input_tokens,
        "cache_read_tokens": response.usage.cache_read_input_tokens,
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "total_cost": calculate_cost(response.usage),
        "cache_hit": response.usage.cache_read_input_tokens > 0
    }
    logger.info(json.dumps(metrics))
 
def calculate_cost(usage):
    # Anthropic pricing
    cache_write_cost = usage.cache_creation_input_tokens * (3 / 1_000_000)
    cache_read_cost = usage.cache_read_input_tokens * (0.30 / 1_000_000)
    input_cost = usage.input_tokens * (3 / 1_000_000)
    output_cost = usage.output_tokens * (15 / 1_000_000)
    return cache_write_cost + cache_read_cost + input_cost + output_cost

Dashboarding

Build a dashboard (ideally in D23's Superset interface) that shows:

• Daily cache hit rate trend
• Cost per query over time
• Cache efficiency ratio by database
• Top queries by token count
• Cost savings vs. baseline (non-cached)

Common Pitfalls and How to Avoid Them

Pitfall 1: Caching Dynamic Content

Problem: If you cache user-specific filters, timestamps, or session tokens, the cache becomes nearly useless because every request is different.

Solution: Separate static (schema, instructions) from dynamic (user query, filters, context). Cache only the static portion.

Pitfall 2: Insufficient Cache Warm-Up

Problem: If your first request of the day caches the schema, and then the cache expires (some providers have TTLs), you lose the benefit.

Solution: Implement cache warm-up—pre-populate caches during off-peak hours or when the application starts.

def warm_up_cache():
    for database in databases:
        schema = fetch_schema(database)
        instructions = fetch_instructions()
        examples = fetch_examples()
        
        # Make a dummy request to populate the cache
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=10,
            system=[{"type": "text", "text": schema + instructions + examples, "cache_control": {"type": "ephemeral"}}],
            messages=[{"role": "user", "content": "SELECT 1"}]
        )
        logger.info(f"Cache warmed for {database}")

Pitfall 3: Over-Caching

Problem: If you cache too much dynamic content in an attempt to maximize cache size, you reduce cache hit rates because the cached content changes frequently.

Solution: Cache only content that's truly static or changes infrequently. The sweet spot is typically schema + instructions + base examples (5,000–10,000 tokens).

Pitfall 4: Ignoring Cache TTL

Problem: Different providers have different cache lifetimes. Anthropic caches for 5 minutes by default; OpenAI for up to 24 hours. If you don't understand the TTL, you might think the cache is working when it's actually expiring.

Solution: Check provider documentation and monitor cache hit rates. If hit rates drop unexpectedly, investigate TTL.

Pitfall 5: Not Accounting for Output Token Costs

Problem: Prompt caching reduces input token costs, but output tokens (the generated SQL) are still full price. If your queries are generating very long outputs, the savings are less dramatic.

Solution: Optimize query generation to produce concise SQL. Use output constraints in your prompt: "Generate the most efficient SQL query in under 500 tokens."

Comparing Prompt Caching to Other Cost-Reduction Strategies

Prompt caching isn't the only way to reduce LLM costs for text-to-SQL. Here's how it compares:

Caching vs. Fine-Tuning

Fine-tuning involves training a smaller model on your specific domain (SQL generation). It reduces inference costs by using a cheaper model.

• Cost reduction: 40–60% (from cheaper model)
• Implementation time: 2–4 weeks
• Latency: Often faster due to smaller model
• Flexibility: Less flexible; requires retraining for new schemas

Prompt caching:

• Cost reduction: 80–90%
• Implementation time: 1–2 days
• Latency: 75% faster
• Flexibility: Highly flexible; no retraining needed

Verdict: Caching is faster to implement and more cost-effective. Fine-tuning is complementary and worth exploring after caching is live.

Caching vs. Query Deduplication

Query deduplication means caching actual SQL query results and returning cached results for identical queries.

• Cost reduction: 0% on LLM costs (you still generate the query)
• Benefit: Reduces database load and query latency

Prompt caching:

• Cost reduction: 80–90% on LLM costs
• Benefit: Reduces LLM API costs and latency

Verdict: These are complementary. Use both: prompt caching for LLM costs, query result caching for database load.

Caching vs. Switching to a Cheaper Model

Cheaper models like Claude 3 Haiku or GPT-4o Mini cost 80–90% less than flagship models.

• Cost reduction: 80–90%
• Implementation time: 1 day (just change the model ID)
• Trade-off: Lower accuracy, may require more examples

Prompt caching:

• Cost reduction: 80–90% (with flagship models)
• Implementation time: 1–2 days
• Trade-off: None; you get the same accuracy

Verdict: You can combine these strategies. Use a flagship model with caching for the best accuracy and cost. Or use a cheaper model with caching for extreme cost reduction (95%+).

The Path Forward: Implementing Caching in Your System

Here's a step-by-step implementation roadmap:

Week 1: Foundation

1. Choose your LLM provider (Anthropic recommended for highest discount)
2. Audit your current text-to-SQL prompts; identify static vs. dynamic content
3. Implement basic caching with Anthropic or OpenAI SDK
4. Set up logging and metrics collection

Week 2: Testing

1. Run A/B tests: caching vs. non-caching on a subset of traffic
2. Measure cache hit rates, cost per query, latency
3. Verify SQL quality hasn't degraded
4. Identify edge cases (schema changes, permission changes)

Week 3: Rollout

1. Deploy caching to 50% of production traffic
2. Monitor metrics for 48 hours
3. Roll out to 100%
4. Document cache invalidation procedures

Week 4: Optimization

1. Implement cache warm-up
2. Fine-tune cached content (remove unnecessary examples, compress schema)
3. Set up dashboards in D23 or your BI platform
4. Plan for schema versioning and cache management

Ongoing

1. Monitor cache hit rates weekly
2. Adjust cached content based on usage patterns
3. Explore advanced strategies (multi-tier caching, cross-database caching)
4. Consider complementary optimizations (query deduplication, cheaper models, fine-tuning)

Conclusion: 80% Is Real, and It's Within Reach

The 80% cost reduction in prompt caching for text-to-SQL isn't aspirational marketing—it's physics. When 80–90% of your prompt is static schema and instructions, and you're caching at 90% discount rates, the math is straightforward.

For data and analytics leaders, engineering teams, and platform builders embedding text-to-SQL into dashboards, self-serve BI, or analytics products, prompt caching is the single highest-ROI optimization available. It requires minimal engineering effort, delivers immediate cost savings, and improves user experience through faster queries.

The path is clear: structure your prompts to separate static from dynamic content, implement caching with your LLM provider, monitor the results, and iterate. Within a month, you'll see 80%+ cost reductions on LLM inference. Within three months, you'll have optimized the implementation to the point where text-to-SQL is no longer a significant cost driver.

For teams using D23's managed Apache Superset platform, prompt caching integrates naturally with embedded analytics and self-serve BI workflows, amplifying the cost and latency benefits. Whether you're a startup scaling query generation, an enterprise standardizing analytics across teams, or a PE firm tracking portfolio KPIs, caching is the lever that makes text-to-SQL economically viable at scale.

Start with the basics. Measure your savings. Then optimize. The 80% is waiting.