Keep containers waiting and 45 seconds becomes a fraction of one — while they idle all day

The previous post ended on this: isolation was not the mistake, waiting for the user before preparing was. So prepare earlier.

There is a ready-made analogy: the taxi rank at an airport. You walk out of the terminal and get straight into a car because the cars were already queued there — nobody dispatched a driver from downtown after you called.

The code: three parts

One, a row of waiting containers. Keep five of them running at all times:

spec:
  replicas: 5          # five standing by, always

Two, each container reports whether it is free.

is_busy = False        # am I working right now

@app.get("/health")    # someone asks, I answer: idle or busy
def health_check():
    return {"status": "busy" if is_busy else "idle"}

Three, a dispatcher. A request arrives, it asks around, and hands the work to the first free one:

for pod in pods:                       # ask each in turn
    resp = await http_client.get(f"...{pod}/health")
    if resp.json().get("status") == "idle":
        return pod                     # found a free one, give it the work

No container gets built on the request path any more, and nothing nags the management system about whether it is ready. The wait drops from 45 seconds to a fraction of a second — how small depends on the machine and the pool size, and the endpoint returns the figure it just measured, so the number that counts is the one from your own run.

Hundreds of times faster, with less code. That rush is deceptive, which is why the second half of this post matters more.

The invoice: those hundreds of times are rented

A taxi rank offers instant pickup because a row of cars is sitting there empty. Empty cars carry nobody, and the drivers are still there, the fuel still burns, the spaces stay occupied. A warm pool is exactly the same trade: money spent continuously, so that users never wait.

Which turns the problem from engineering into business:

• how many cars are enough? Not average demand, peak demand. An empty pool answers “no capacity available right now”, which is a worse experience than waiting 45 seconds;
• how do you adjust to demand? Adding cars means paying that 45 seconds again, so scaling has to happen ahead of the peak rather than in reaction to it;
• can you keep fewer? Yes, and the pool empties more easily at peak. You can also drop to zero when it is quiet, and the first customer pays the 45 seconds again.

There is no standard answer, only the position your business can live with. What matters is that the position is chosen deliberately — the danger of a warm pool is precisely that it makes the latency problem look solved.

Three things this dispatcher gets wrong

The code is in the repo, and so are its flaws, because they are exactly what forces the next posts.

One, there is a gap between asking and assigning. The dispatcher asks “are you free”, hears yes, then sends the work — with a network round trip in between. Under load two requests can pick the same container; the slower one is told “I am already busy”, and since the dispatcher does not retry, that error goes straight to the user.

Two dispatchers assign the same empty taxi to two passengers. The first one gets in, the second is left standing there, and nobody calls them another car.

Two, asking around costs more as you grow. Invisible at five containers; at fifty it is up to fifty network round trips, each with a timeout to wait out. “Find a free one” becomes its own source of delay.

Three, when full it can only refuse. There is no notion of a queue or a ticket number, so a peak is absorbed by turning people away.

One implementation shortcut deserves naming: the dispatcher reaches containers through a local kubectl proxy tunnel. Fine for local development, not a production pattern — production would use service discovery or address the containers directly.

The next wall

All of the above is survivable. What is not: “am I busy” lives, like task progress, only in that container’s own memory.

When the container dies, both go with it. And these containers get reclaimed, rolled and taken down with their machines routinely — which is exactly what the next post takes apart.

In one line: the speed is rented with permanent idling; that trade has to be priced on purpose, not discovered when the bill arrives.

Code: agent-sandbox-oss/lab3.