Fleet runners (runs_on)
Peregrin nodes can advertise runner labels and campaigns can declare which labels they require. Scheduling rides Hypha's entity plane — campaign entities, campaign.start ops, ownership and locks — never a separate lease protocol.
Node labels
PEREGRIN_RUNNER_LABELS is a comma-separated list appended to the node's /api/node capabilities (alongside ephemeral, sessions, …):
PEREGRIN_RUNNER_LABELS=runner:linux,gpu peregrin serve --ephemeral
PEREGRIN_RUNNER_LABELS=runner:linux,gpu peregrin campaign serve docs-deploy.yamlThe hub's sync projection persists the advertised capability list, so fleet members are discoverable via node status.
Campaign runs_on
name: gpu-train
runs_on: [runner:linux, gpu]
nodes: [...]Every runs_on entry must appear in the node's advertised capabilities (AND semantics; absent runs_on runs anywhere). The gate is enforced in two places:
campaign.startentity ops — the campaign entity plane on the owning node rejects a start whoseruns_onthe node doesn't satisfy (409), so a misrouted op fails loudly instead of running on the wrong hardware. Combined with Hypha ownership + locks, only the entity's owner can launch it — two nodes can't claim the same campaign entity.campaign servehosting — campaigns whoseruns_onisn't satisfied are skipped at startup (warned); if none match, serve exits with an error rather than hosting half a fleet.
Job isolation
driver: shell params accept:
isolate: true— run the command insidesystemd-run --user --scope --collect.memory_max: "512M"— adds-p MemoryMax=…and impliesisolate.
Both require systemd-run on PATH and a working systemd --user manager; without them the node fails with node.failed — isolation is never silently dropped. bupha-imported jobs carry isolate / memory_max through to the inner shell driver automatically.
Deferred: cgroup v2 direct manipulation, container/VM isolation, per-job CPU/IO limits beyond MemoryMax.
Ephemeral-node recovery
An ephemeral runner (--ephemeral / PEREGRIN_EPHEMERAL) that dies mid-run is reaped by the hub after PEREGRIN_EPHEMERAL_DEAD_AFTER: its projected entities drop and outbox writes queued for it dead-letter. Campaign entities hosted by campaign serve keep their durable SQLite queue — on restart, interrupted runs requeue at the front per the queue's at-least-once semantics, and supersede keys still dedupe retriggered work.
Hub-side dispatch
When a campaign entity is created through the hub without an explicit node pin (POST /api/entities, no node field or ?node=) and its declared runs_on labels are non-empty, the hub's FleetDispatchApi decorator picks an eligible node automatically:
- Eligible = node
status != "down"andruns_onsatisfied over the node's persistedcapabilitieslabels. - Sort order is
(ephemeral, id)— durable nodes are preferred, ephemeral runners are the fallback; ties break on node id, so the pick is fully deterministic. - The created entity is then owned by the picked node, so
campaign.startops route there and pass the existing label gate.
Callers can still pin explicitly (node field / ?node=); pinned or label-less creates pass through untouched. Known seams: moving an entity already pinned to an ineligible node is snapshot export/import territory; there is no re-dispatch on node death and no load-aware scheduling yet.