System Design LiveHands-on challenges

Token Rate Limiter

All challenges

Token Rate Limiter

Five teams call the model through your gateway. The model provider gives the company one shared quota: 72,000 tokens a minute. Over it, the provider answers 429 Too Many Requests to whoever asks next.

The gateway already limits each team to 120 requests a minute. Four teams send small prompts, one request a second each. The fifth runs a batch job: one request a second too, but each prompt is 3,000 tokens.

Now the four small teams keep getting 429s, and none of them is anywhere near its request limit.

System Architecture

4 small + 1 bulkone shared quota5 tenantsLLM GatewayLLM provider(72k tokens/min)

Your task

Keep the small teams' calls flowing, and give the batch job only its fair share.

You edit only app.py in the gateway repo, already cloned in the editor. Start with the Limiter class.

  • The provider counts tokens, not requests: a request costs its prompt plus max_tokens. request_tokens(body) already computes that.
  • Each team's fair share is TENANT_TPM: 14,000 tokens a minute. Five shares fit inside the provider's 72,000.
  • When you refuse a request, answer 429 with a Retry-After header saying how many seconds to wait.
  • Commit + push → the platform runs the tests, rebuilds the image and redeploys the gateway.

Success Criteria

When you submit, the Judge watches the next 60 seconds of live traffic. All three must pass:

  • Small tenants refused or failed: 0.
  • Bulk tenant's prompt tokens per second, counted by the provider: between 100 and 400. Its fair share is about 233.
  • Bulk tenant: refusals without Retry-After, or errors: 0.

Where to look

  • Provider 429s / s: the provider refusing everyone, because the gateway let too much through.

  • Bulk tenant: prompt tokens / s at the model: how much of the quota one team is using.

  • Terminal: ask the gateway yourself, as any tenant:

    bash
    curl -si gateway/v1/chat/completions -H 'Authorization: Bearer me' -d '{"messages":[{"role":"user","content":"hi"}],"max_tokens":5}'

Submit once small tenants have had no refusals for a full minute.

  • 120 requests a minute means nothing when one request is 15 times bigger than another. Count what the provider counts.
  • A token bucket per tenant: it holds up to TENANT_TPM tokens, refills TENANT_TPM / 60 every second, and each request takes request_tokens(body) out of it.
  • When the bucket is short, the wait is how long it takes to refill the difference: (cost - tokens in the bucket) / refill rate.
  • Turning the request limit down doesn't fix it: a limit low enough to slow the batch job refuses the small teams too.