Stream the Tokens

This service answers questions about products. The model writes each answer one token at a time, about 150 tokens over 3 seconds.

The API already answers as a stream of events, one per token. But users stare at a blank screen for 3 seconds, and then the whole answer appears at once.

System Architecture

Your task

Get the first token to the user as soon as the model writes it.

You edit only app.py in the chat-api repo, already cloned in the editor. Start with chat().

The response format stays the same: one data: {"token": "..."} event per token, then data: [DONE].
The answer must be exactly what the model wrote. The grader rebuilds it from the events and checks every character.
Commit + push → the platform runs the tests, rebuilds the image and redeploys the API.

Success Criteria

When you submit, the Judge watches the next 60 seconds of live traffic. Both must pass:

Failed or wrong answers: 0.
Time to first token, p95 ≤ 0.4 s. The starter takes about 3 s.

Where to look

Time to first token vs Full answer: right now they're the same number. When you're done, the first drops to almost nothing and the second stays about the same.

Terminal: watch the model stream by itself:

bash

curl -sN fake-llm/v1/chat/completions -d '{"messages":[{"role":"user","content":"hi"}],"max_tokens":20,"stream":true}'

Submit once time to first token has stayed under 0.4 s for a full minute.

The starter asks the model for the whole answer, then splits it into events. The model can stream: send "stream": true.
requests waits for the whole body unless you pass stream=True too. Then read it with resp.iter_lines().
Flask sends a response as you produce it if you return a generator. Wrap it in stream_with_context.
Building a list of events and returning it at the end is still buffering, however you got the tokens.