Containers die whenever; two hours of progress should not die with them

Three posts in: tasks are isolated from each other, and the wait went from 45 seconds to a fraction of one. But one thing has not changed since the first post — progress still lives in the container’s own memory.

That used to mean “a crash loses it”. Now it is worse: containers in a warm pool are replaced as a matter of routine. Upgrades replace them, machine failures replace them, resource pressure reclaims them. They are consumables by design.

Which reframes the question: why would you let a consumable hold two hours of your work?

Write it somewhere else: the site blackboard, not the worker’s head

Picture a construction site. Progress kept in one worker’s head ends when the shift changes; progress chalked on a blackboard on the wall can be read by whoever comes next.

The code does exactly that — hang the blackboard on the wall:

# ask for a 1GB shared disk
kind: PersistentVolumeClaim
spec:
  resources:
    requests:
      storage: 1Gi
---
# mount it into every container in the pool
volumeMounts:
  - name: workspace-volume
    mountPath: /workspace     # this is the directory the container sees

In K8s a shared disk like this is a persistent volume. The operative word is persistent: it outlives the container, and the next container can mount it.

Then two changes to the work itself. After each page, write it on the board:

for i in range(start_page, total_pages):
    await asyncio.sleep(1.0)                                  # fetch a page
    with open(f"/workspace/{task_id}/checkpoint.json", "w") as f:
        json.dump({"pages_crawled": i + 1, "total": total_pages}, f)   # write immediately

Before starting, read what the last shift left:

start_page = 0
if os.path.exists(ckpt_file):
    start_page = json.load(open(ckpt_file)).get("pages_crawled", 0)
    print(f"checkpoint found, resuming from page {start_page}")

That is all of it. No distributed transactions, no compensation logic — one JSON file.

Now delete the working container on purpose

This step can only be judged by breaking something; reading the code proves nothing.

Halfway through a task, delete the container doing the work:

kubectl delete pod <the-one-currently-working>

Re-dispatch the same task and watch the new container’s log:

checkpoint found, resuming from page 17
[worker-b] task-a3f9 progress: 18/30
[worker-b] task-a3f9 progress: 19/30

The first seventeen pages are not redone. The container changed; the work continued where it stood.

One easily-missed consequence: the progress endpoint can now ask any live container, because they all mount the same disk. The state no longer belongs to a particular container.

This is what the often-quoted line actually feels like: containers are cattle, workspaces are pets. Cattle are replaceable; pets have names and need looking after.

What this version still lacks (the honest part)

One, the “automatic recovery” needs a human. The code says so plainly:

# no task_id means a new task; a task_id means resume

In other words, nothing is watching for tasks that stopped. Someone has to notice, then re-send the request with the original task ID. That is not automatic failover, it is “resumable” — and the distance between the two is an entire monitoring and retry mechanism. Closing that gap is what the next two posts are about.

Two, that disk will not mount on three containers in a real cluster. It is requested as ReadWriteOnce, meaning one machine at a time. The local experiment has exactly one machine, so three containers sharing it is fine; on a real multi-machine cluster, containers on the second machine fail to mount. Genuine sharing needs storage that supports multi-machine access — NFS, or a cloud file service.

Three, writing after every page is a trade-off, not a default. Writing more often loses less on a crash and costs more in I/O. In a real system that frequency should be derived from how expensive it is to redo one unit of work.

What this step settles

Once progress leaves the container, the container really is a replaceable executor. But notice what is now missing: nothing in the system knows whether a task has stalled — the dispatcher hands work over and forgets it.

And when the pool is full, it still only knows how to say “no capacity available”.

Those two are the same gap: there is no place that remembers what work is still outstanding. The next post fills it — and, as it turns out, that is where I discovered I had not actually built the thing I thought I was building.

In one line: let the state outlive the container, and the container is free to die.

Code: agent-sandbox-oss/lab4.