A queue turned rejection into waiting — then I noticed I was still pushing work

The previous post left two holes: nothing remembers what work is outstanding, and a full pool can only say “no capacity available”.

They are the same hole. The system has no list of things waiting to be done.

The ticket machine at the bank

Physical queues solved this long ago: take a number at the door, sit down, wait to be called. The ticket machine does not care which counter is free — its only job is to put you in line.

In code that is a queue. The dispatcher’s job collapses to almost nothing — accept the task, put it in the queue, tell the user their position:

r.lpush(QUEUE_NAME, json.dumps(task_data))   # into the queue, that is the whole thing

return {
    "message": "task queued",
    "position_in_queue": r.llen(QUEUE_NAME), # how many are ahead of you
}

It no longer needs to know how many containers exist, or which of them is busy. At peak, users stop seeing “no capacity available” and start seeing “you are seventh in line” — the same waiting, an entirely different experience: turned away at the door, versus already inside the queue.

At the other end, a scheduler process watches the queue:

while True:
    if r.llen(QUEUE_NAME) > 0:                   # work is waiting
        worker_name = await find_idle_worker()   # find a free container
        if worker_name:
            task = r.rpop(QUEUE_NAME)            # take one task out
            await dispatch_task(worker_name, task)   # hand it over
    await asyncio.sleep(1)

It works: nothing is lost, peaks queue up, users get a position instead of an error.

Then I read that code back

I thought this step changed the system from pushing work to pulling work — the textbook move: instead of a central component shoving a task at a chosen worker, whoever becomes free goes and takes the next task themselves.

But look at what that code does. find_idle_worker() asks each container whether it is free, then dispatch_task() delivers the work.

That is still pushing. All I did was move the pusher from the front desk to the back office.

Against the bank analogy: I meant to build “a counter finishes and calls the next number”. What I built is “a floor manager asks each counter whether it is busy, then walks the customer over”. The difference is not stylistic:

	Floor manager assigns (what I built)	Counter calls next (what it should be)
Can two get the same task	Yes — there is a gap between asking and assigning	No — taking from the queue is exclusive by itself
Must every counter be polled	Yes, and it gets slower with more of them	No, whoever is free simply comes
Must the centre know who is idle	Yes	No, the centre only owns the queue

So the two flaws from post three — the race between asking and assigning, and the cost of polling everyone — are both still there. They merely moved out of the user’s sight into the background. The user no longer sees the error; the problem has not moved.

Real pulling takes one blocking read by the worker itself (BRPOP in Redis): whoever is free goes and takes a task, and taking it makes it theirs — nobody else can take the same one. No lock, no table of who-is-idle, no polling from a central process.

I wrote this section instead of quietly fixing the code because it is the most useful mistake in the series: “we added a queue” and “we moved to a pull model” are two different claims. The first buys buffering; only the second eliminates scheduling contention. I had assumed the first delivered the second.

One more thing that is not implemented

The submission endpoint accepts a field:

task_data = {"task_id": task_id, "pages": pages, "is_vip": is_vip}

is_vip travels all the way into the queue and then — nothing. The scheduler has exactly one first-in-first-out queue, and serves it in order. Priority: field accepted, logic never written.

Leaving room in an interface is not wrong in itself, but unmentioned it reads like a feature that exists. Doing it properly usually means two queues, VIP checked before standard — and then a rule that stops the standard queue from starving.

What this step settles

What it genuinely bought: peaks turn from rejection into queueing, and tasks stop being lost. The dispatcher and the workers are now fully decoupled — the dispatcher does not even know how many containers there are.

What it did not buy: scheduling contention is still in there, only hidden.

The last post takes every piece built so far, puts it under load, and separates what is genuinely stable from what merely has not been pushed hard enough — and explains why the hard-isolation step is where I stopped at theory.

In one line: a queue buys buffering, not freedom from locks; handing the “who does it” decision to the edge is the move that actually removes work.

Code: agent-sandbox-oss/lab5.