Nginx Load Balancing: Fix an Upstream Stuck on One Server

You have an API behind nginx and two identical backend servers. You check the traffic, and every request is landing on server one. Server two hasn't seen one request.

You've paid for two servers and you're getting the capacity of one. This is a common nginx load balancing mistake. The fix is one upstream block, but one plausible-looking choice puts you right back where you started.

This walkthrough is based on a hands-on challenge from System Design Live. System Design Live drops you into a real, broken production system in your browser — you fix it, and a judge grades your fix against live traffic. Here's the problem, the fix, and the trap.

The setup

A client sends HTTP traffic to an nginx reverse proxy on port 80. Behind nginx are two identical Flask backends, flask-app-1 and flask-app-2, which identify themselves as server-1 and server-2.

The nginx config proxies requests to an upstream called flask_backend, and that upstream lists only one server:

upstream flask_backend {
    server flask-app-1:80;
}

Nginx is doing what it was told. It has never heard of flask-app-2.

Why this hurts

A load balancer can only spread work across backends it knows about. If the upstream lists one server, the "load balancer" is just a proxy with a single destination, and the second backend is capacity you're paying for and not using.

In this system the cost is real. The /process endpoint is expensive on purpose: at least 1.5 seconds of I/O plus real CPU work per request. When all of that lands on one backend, requests pile up behind each other. CPU hits its limit, the queue grows, and latency climbs. Meanwhile an identical backend sits idle.

That's what load balancing is for: spreading work so that no single box becomes the bottleneck. You won't fix this by adding a third server or making the backend faster. You fix it by making sure the traffic reaches the servers you already have.

How to tell it's broken (and fixed)

Don't guess. You have two signals, and both of them tell you about distribution:

Logs. These are the most direct check. The client simulator prints how many requests each backend has handled:

backends=map[server-1:412 server-2:405]

When it's broken, server-2 is missing or stuck at zero. When it's fixed, both names appear and both counts keep climbing.

Metrics. Look at the request rate for each backend. Before the fix, server-1 takes every request and server-2 stays flat at zero. After the fix, both lines rise together. Watch the request distribution first — it's the clearest signal and the one the judge grades. Resource usage follows it:

The fix

Add the second backend to the upstream:

upstream flask_backend {
    server flask-app-1:80;
    server flask-app-2:80;
}

Note the port. The backends are addressed by their Kubernetes Service name on port 80. Each Service forwards that traffic to the Flask container on port 8000, but nginx talks to the Service, so it uses :80. If you point nginx at :8000, you get connection failures instead of load balancing.

With the second backend in the upstream, the architecture becomes:

And the load moves accordingly:

Next, ship it. In this system the platform rebuilds the nginx image and rolls out the new config only after you commit and push the change. Check the deployment logs. A syntax error in nginx.conf stops nginx from starting, so the logs are where a typo shows up.

Choosing an nginx load balancing strategy

Listing both servers is enough to get traffic moving. How nginx splits that traffic depends on the strategy you choose in the upstream block.

Round robin (the default)

upstream flask_backend {
    server flask-app-1:80;
    server flask-app-2:80;
}

If you don't name a strategy, nginx uses round robin and sends new requests to each backend in turn. It needs no extra configuration, and under this challenge's traffic pattern it's enough to pass.

least_conn: a good fit for slow, uneven requests

upstream flask_backend {
    least_conn;

    server flask-app-1:80;
    server flask-app-2:80;
}

least_conn sends each new request to the backend that currently has the fewest active connections. That helps when requests take different amounts of time. A backend stuck on a few slow requests gets fewer new ones, and the less busy backend picks up the slack. Because /process is expensive and uneven, least_conn is a natural fit here.

The trap: ip_hash

upstream flask_backend {
    ip_hash;

    server flask-app-1:80;
    server flask-app-2:80;
}

This config looks like a load balancer: two servers and a strategy keyword. But ip_hash doesn't balance load. It provides source-IP affinity: it hashes the client's IP address, and every request from that IP goes to the same backend.

That's useful when you need a client to stick to one server. In this system it's fatal, because all simulator traffic comes from a single client IP. ip_hash hashes that one IP, chooses one backend, and sends everything there. You're back to one server doing all the work and the other sitting idle, and now your config makes it look like you've fixed it.

Affinity is not the same as distribution. Before you reach for ip_hash, ask how many distinct client IPs you actually have. Behind a single upstream proxy, a NAT gateway or a test harness, the answer can be "one".

Common pitfalls

You edited the file but didn't push it. The platform rebuilds nginx only after you push. If your change seems to have no effect, check the repository state and git log in the Dev VM.

You used the wrong port. Use flask-app-1:80 and flask-app-2:80, not :8000. Port 80 is the Service that nginx can reach. Port 8000 is the Flask container port behind it. The wrong port means connection failures and failed requests.

You submitted too soon. The graded window opens when you click Submit, and anything before that isn't scored. Wait until the new nginx deployment is live and the logs show both backends serving, then submit and let the full window run.

What "fixed" means

When you submit, the judge watches the next 60 seconds of live traffic. Three checks must pass:

• Zero failed requests. No 4xx/5xx responses, connection failures or timeouts. This also proves your nginx config is still valid.
• Server 1 serving. Traffic keeps reaching server-1 for the whole window.
• Server 2 serving. This is the check the challenge turns on. It reads zero until traffic actually reaches the second backend, and one request isn't enough. Traffic has to keep arriving for the whole window.

An ip_hash config can pass the first two checks and fail the third, which is exactly why the trap matters.

Try it yourself

Reading the fix and shipping it are different things. In the challenge, you edit the real nginx.conf, push it, watch the per-backend metrics move, and a judge grades your fix against 60 seconds of live traffic. It's an easy one, so it's a good first rep.

Try the Nginx Load Balancer challenge free, no setup →