Shard the Writes (Router)

All challenges

Shard the Writes (Router)

Your order service has outgrown one Postgres database, so the team has added a second. shard-0 holds all 1M existing orders and serves all the requests. shard-1 is empty.

Splitting the traffic is only half the job. The moment your router sends a customer to shard-1, their existing orders are still sitting on shard-0, and every read for them comes back 404. The data has to move too.

System Architecture

orderseverythingClient SimulatorOrder APIshard-0 (1M)shard-1 (empty)

Your task

Shard the orders across both shards by customer_id, and move the existing orders that now belong on shard-1, without losing a single one.

Downtime is acceptable. Take your time: put the order API in maintenance mode, split the data between the shards, deploy the router, then bring traffic back. The judge only grades the minute after you submit.

You edit the order-api repo, already cloned in the editor. Every commit + push runs the tests, rebuilds the image and redeploys the order API. A failing test fails the build, so check the build log if your change doesn't show up.

1. Maintenance mode. In config.py, change the MAINTENANCE_MODE default from "false" to "true" and push. Once the new pod is up, every order request answers 503 and nothing writes to the shards. Watch 503s / s climb before you move any data.

2. The migration. From the editor's terminal, which has psql:

  • Connect to a shard with psql postgresql://orders:orders@shard-0:80/orders (use shard-1 for the other one). The port is 80, not 5432.
  • Copy the orders your router will send to shard-1 across, then delete them from shard-0.
  • Move rows with \copy, not a plain SELECT. psql holds a plain SELECT's whole result in memory, and 500k rows crash the editor.

3. The router, and traffic back on. In one push:

  • Write shard_for(customer_id) at the top of orders.py: which shard owns this key? Use db.shard_count(), not a hardcoded 2.
  • orders.py hardcodes shard = 0 in create, get, update and delete, each with a numbered TODO. All four must ask shard_for().
  • Set the MAINTENANCE_MODE default back to "false".

Success Criteria

When you submit, the Judge watches the next 60 seconds of live traffic. All three must pass:

  • No lost orders: every order that exists reads back, new ones and existing ones alike.
  • No response errors: no failed requests. That includes 503s, so maintenance must be off.
  • Shard balance: no shard holds more than 60% of the orders, counted in the databases.

Submit once lost orders are at zero, the 503s have stopped, and both shards hold about half the orders, for a full minute.

  • Order matters: maintenance on, move the data, then push the router. If the old router keeps writing while you copy, new orders land on shard-0 after your copy and are stranded there.
  • A push takes a minute to roll out, and the old pod keeps serving until the new one is ready. Start copying only once 503s / s is up.
  • Your migration has to pick exactly the rows your router sends to shard-1. customer_id % 2 is easy to write in both Python and SQL; a hash only your Python code knows is not.
  • psql's \copy (SELECT ...) TO STDOUT on one shard piped into \copy orders FROM STDIN on the other moves rows without a temp file. Then delete the moved rows from shard-0.
  • Every path (create, get, update, delete) must make the same routing decision. A read sent to the wrong shard isn't an error: it's a quiet 404.
  • Route by customer_id, never the order id. The reader only has the customer id.
  • Balance the orders, not the id range: a few hundred customers place most orders, so customer_id < 50000 still leaves one shard with most of them.