Usage Metering

Your platform resells model access. Three tenants stream completions through your metering proxy, and each is billed for the tokens they use, recorded in Redis.

The model provider bills you for what it actually generates. You bill tenants an estimate made before each call: the prompt plus max_tokens. Your invoices come to about twice the provider's, and the tenants have noticed.

System Architecture

Your task

Bill each tenant exactly what the model actually generated for them.

You edit only app.py in the metering-proxy repo, already cloned in the editor. Start with chat().

The stream's last chunk, just before data: [DONE], carries usage.total_tokens: what the model billed for the request.
Some readers lose interest and hang up part-way through a long answer. The model stops when they do, and bills only what it sent.
prompt_tokens(body) counts the prompt exactly as the model does. Each content chunk in the stream is one token.
Commit + push → the platform runs the tests, rebuilds the image and redeploys the proxy.

Success Criteria

When you submit, the Judge watches the next 60 seconds of live traffic. Both must pass:

Failed streams: 0.
Tokens billed to tenants / tokens the model billed: between 0.95 and 1.05.

Where to look

Billed by you / billed by the model: 1.0 is the goal.
Abandoned streams / s: the requests whose final chunk never arrives.

Submit once the billing ratio has stayed between 0.95 and 1.05 for a full minute.

max_tokens is a limit, not a prediction. Most answers stop well short of it.
Bill after the stream, from what came back. The last chunk has the exact number.
When a client hangs up, there is no last chunk. Count the prompt plus every content chunk you relayed, and bill that.
A finally: in the generator runs however the stream ends, finished or cut off.