A fleet of 38 AI agents that keeps running when nobody watches

The problem

An AI agent in a chat window does what you ask while you watch. That is not the same as AI doing work. The gap is unglamorous: what happens when nobody is watching.

Every real failure I hit was operational, not a question of intelligence:

• The machine reboots. Every agent is gone, and nothing notices.
• An agent goes idle with work still queued. It finished its task and now waits to be spoken to.
• An agent crashes but its session survives. Every health check reports green while the agent is dead. This is the expensive one.
• An agent dies holding a claimed job. The job stays “in progress” forever and no other agent touches it.
• A task needs credentials. Now the blast radius of one bad instruction is every client you have.

None of this is fixed by a better prompt. It is ordinary systems reliability work, and in my experience it, not model capability, is what keeps companies from putting agents into production.

The approach: queue, watchdog, a real definition of “alive”

Dispatch through a queue, not through attention. Tasks arrive via Slack and my own message queues (Node.js + SQLite), tagged with the agent that owns them. Agents claim, work, respond. A scheduled job nudges each session so an idle agent picks up queued work without anyone prompting it. Producing and consuming work are fully decoupled: I can queue twenty tasks at midnight and go to bed.

A supervisor with an opinion about what “alive” means. A control tool covers status, ensure-all, restart, a pane dump for when an agent goes quiet, and a doctor command for config drift and scheduler coverage. The watchdog runs every minute and at boot. A crashed agent is rebuilt automatically, with a guard against restart loops: at most 3 restarts in 30 minutes.

The key design decision: liveness is not “the session exists”. That check is exactly what made crashed agents invisible. The supervisor inspects what is actually running inside the session, because the failure I most needed to catch was the one that looks healthy.

$ agentctl doctor
── registry drift ───────────────────────
  ✓ registry matches disk
── keepalive coverage ───────────────────
  ✓ all agents covered
── recovery jobs ────────────────────────
  ✓ @reboot recovery installed
  ✓ periodic ensure installed
── unmanaged sessions (informational) ───
  ℹ  ad-hoc sessions, not supervised
✓ doctor: no problems found

New client agents are set up by script in minutes. Around 30 supporting services run under pm2.

Isolation by blast radius

Separation by machine. Client work runs on the always-on server, which holds the client credentials. Experimental work runs on the GPU workstation with its own queue and its own database, and the supervisor configuration there refuses client agents by design. Both fleets run with permission prompts disabled, which is precisely why the boundary is enforced by the machine and not by good intentions.

Stuck-job recovery, tuned per machine. A reaper releases jobs held by an agent that died. On the GPU box its timeout is longer, because a transcription batch or a render legitimately runs long, and re-dispatching a live job wastes the GPU it was waiting for.

Porting it: three failures that looked identical

I built the supervisor on the server and ported it to the workstation. It failed three times, and every failure looked the same: session exists, agent does nothing.

1. The trust prompt was inverted. One machine pre-selected “Yes, I trust this folder”, the other listed “No, exit” first. The supervisor answered by keystroke position, so on the second machine it confidently chose No. The agent quit, the session survived as a plain shell, and every health check stayed green. Fix: select the option by its label, which also survives the next upstream reordering.
2. Launching in a directory is not being in it. The shell profile on the workstation dropped each new session back into the home directory. The agents opened the wrong workspace, found no specification file and ran with no instructions or boundaries at all. The most dangerous of the three, and the quietest. Fix: an explicit directory change before launch.
3. A health check that always warns is one nobody reads. I merged one keepalive script per agent into a single parameterised script. The doctor command recognised agents by their name in the scheduler line, and the generic script names none of them, so every agent warned permanently. A check that always fires trains you to ignore it, which is worse than no check.

An agent that is silently misconfigured is worse than one that is plainly dead. A dead agent gets restarted. A misconfigured one keeps accepting work.

Outcome and where it applies

Today 38 Claude Code agents run on two machines (31 on the server, 7 on the GPU workstation under WSL2), across 57 tmux sessions in total. 18 of them are client expert agents, one per agency client, each with its own CLAUDE.md and knowledge files. They do real work every day and recover from reboots and crashes without intervention. I can also operate them on the go via Claude Code remote control.

The hard parts were reliability, isolation and observability, not the model.

This applies to any team that has proven an AI workflow in a chat window and now needs it to run unattended, on a schedule, at volume, without someone babysitting it, and to know when it breaks instead of hearing it from a client. It applies just as directly wherever agent work must be kept away from production credentials, which covers most regulated environments.

Technology