The 3 AM Incident

All challenges

The 3 AM Incident

03:12 AM. Your phone goes off:

text
[PAGE] payments-api: success rate 0%, ALL payments failing

Someone deployed PAY-1847: Make worker count configurable at 23:47 and went to bed. Now every payment is failing, and it's your problem.

This challenge starts broken. There's nothing to build, only a production incident to end. Find it, fix it.

System Architecture

POST /payCustomer TrafficPayments APIDown

Your task

Restore payments: find out why payments-api is down and fix it.

You edit the payments-api repo, code and config, already cloned in the editor:

  • The Service forwards payments-api:80 → 8000, and the readiness probe calls GET :8000/health.
  • Commit + push → the platform rebuilds the image and redeploys the payments API.

Success Criteria

When you submit, the Judge watches the next 60 seconds of live traffic. Both must pass:

  • Payments are flowing: a steady rate of successful POST/pay responses.
  • No failed payments: no errors, refused connections or half-recovered flapping.

Where to look

  • Logs: payments-api is your crime scene. What does the service say when it starts, or tries to?
  • Metrics: is the pod even running? A restart loop has a very particular heartbeat.

Submit once customer payments have been green for a full minute. Submissions are free, so your best score counts.

  • Read the traceback like it's pointing at something. It is.
  • "It starts fine now" and "it's serving traffic" are different claims. Verify the second one.
  • The platform contract is written down in README.md in the repo. Compare it with what the service says it's doing.