Prefix Cache Order

Prefix Cache Order

This service is a support chatbot. Every chat sends the model the same 3,000-token support handbook, plus the customer's question.

The model server caches prompt prefixes: if a prompt starts the same way as an earlier one, it skips reprocessing that part. That's faster, and cached tokens cost a tenth of the price.

But nothing is hitting the cache. Every chat takes about 0.7 seconds and pays full price for the whole handbook.

System Architecture

Your task

Make the prompt cacheable, without changing what's in it.

You edit only app.py in the chat-api repo, already cloned in the editor. Start with build_messages().

• Every prompt must still contain the whole handbook, the customer's plan and their question. The grader checks every answer.
• Every chat must still call the model. Don't cache answers in the app.
• Commit + push → the platform runs the tests, rebuilds the image and redeploys the API.

Success Criteria

When you submit, the Judge watches the next 60 seconds of live traffic. All four must pass:

• Failed or wrong answers: 0.
• Chat p95 latency ≤ 0.4 s. The starter takes about 0.7 s.
• Prompt tokens served from the prefix cache ≥ 80%. The starter gets 0%.
• Model calls per chat ≥ 0.9.

Where to look

• Prompt tokens served from the prefix cache and Model spend ($ per chat) are the model server's own numbers.

• Terminal: send the model a prompt yourself and read usage.prompt_tokens_details.cached_tokens. Send it twice:

  bash

  curl -s fake-llm/v1/chat/completions -d '{"messages":[{"role":"system","content":"same text every time ..."},{"role":"user","content":"hi"}]}' | jq .usage
  

Submit once the cache hit rate has held above 80% for a full minute.