Skip to content

Queue & supersede policy ​

The engine executes one run at a time; the supersede policy decides which runs happen when trigger events outpace the executor. This is node-level policy, not engine internals — it lives in peregrin-queue (pure logic, no I/O) and the node daemon drives it around each Runner. The peregrin-node RunSupervisor composes the queue with the engine's ingress channel into the sequential run loop the daemon will own.

supersede: campaign config ​

yaml
supersede:
  key: "{repo}:{ref}"     # template over the trigger event payload
  mode: latest-wins       # | fifo
  cancel_running: true    # cancel the active run vs queue behind it
FieldTypeDefaultMeaning
keytemplate""{dotted.path} placeholders resolved against the trigger event's payload; events resolving to the same key are alternative triggers for the same logical run (one branch, one PR, one WAL head). Empty = a constant key, so every event targets the same run.
modelatest-wins | fifolatest-winsPending-job dedupe: latest-wins replaces the pending job that shares the key (only the newest trigger for a target runs — the displaced job's queue position is inherited); fifo appends (every trigger runs) up to MAX_PENDING_PER_KEY (32) per key — beyond that the key's oldest pending job is evicted.
cancel_runningboolfalsetrue: a new event marks the active run superseded — the queue raises a one-shot cancel signal the caller delivers to the run. false: new events always queue behind the active run.

Key template grammar ​

Distinct from ${...} scope resolution — the {path} grammar is a small payload-only interpolation:

  • {repo}, {origin.ref} — dotted-path walk over event.payload, one key per . segment.
  • Values stringify like JS: null → "null", numbers/booleans as String(v), arrays join with ,, objects → "[object Object]".
  • A miss — absent key, non-object step, empty or unclosed placeholder, an interior containing { — stays verbatim ({missing} in the key is a literal, still deterministic).
  • {{ emits a literal { and rescans: {{repo}} → {a/b}.

Queue semantics ​

Queue::submit(event) resolves the key and returns a Decision (key, disposition, cancel_active):

  • Dedupe applies to pending jobs only — under latest-wins the new event replaces the same-key pending job in place and the displaced Job comes back in Disposition::ReplacedPending(job) for observability. The active run is never deduped: same-key events while it runs queue another run (or cancel it, with cancel_running).
  • Cancel is a one-shot signal: the first cancel_running submit while a run is active reports cancel_active and stashes the superseded Job, handed out once by take_cancel(). finish() clears an unread signal — a run that ended is moot to cancel.
  • Drain is sequential (the gate case): drain() returns None while a job is active and otherwise pops the front job and marks it active; finish() releases the slot. Pending jobs iterate in drain order via pending().

Ingress injection channel ​

External events can't call LocalBus::publish directly — it's &mut and the runner is single-threaded, so a blocking producer (a webhook listener, a WAL poller) would stall the loop (audit I-014). The channel splits that (peregrin-engine, std::sync::mpsc — the engine is synchronous):

  • peregrin_engine::ingress() returns (Injector, Ingress).
  • Injector is the Send + Sync, cloneable producer half — any thread or spawned ingress driver pushes Events without &mut access to anything. inject fails with InjectError (carrying the event back) once the run's receiver is dropped.
  • Ingress is the runner's receiver — Runner::with_ingress attaches it and the run loop drains it into pending once per iteration.

Ordering guarantee: injected events serialize after every event already pending at the drain — including the triggers of the step currently executing — and before anything nodes emit afterward. A workflow.cancel pushed through the injector is honored at the next iteration boundary, exactly like a node-emitted one (it is consumed by the loop, not published — the log shows workflow.cancelled).

Two edge rulings:

  • Events pushed before run() starts are delivered — the runner drains on the first iteration too (RunSupervisor uses this to deliver the trigger event itself).
  • Events arriving while the last step executes still land: the loop drains before its exit check, so only an event arriving after the final iteration can be dropped — a quiescent run exits; the channel does not keep it alive.

RunSupervisor composition ​

peregrin_node::supervisor::RunSupervisor is the composable wrapper the daemon drives: submit(event) → SubmitOutcome { decision, start, cancelled_active }; finish() → Option<PendingRun>.

  • start is Some(PendingRun) only when no run is active — it carries a fresh (injector, ingress) pair for its run with the trigger event already queued on it; hand ingress to Runner::with_ingress, clone injector for external producers.
  • Under cancel_running, a superseding submit pushes Event::cancel("superseded by a newer trigger for key '<key>'") into the active run's own ingress channel — the cancellation surface the single-threaded runner actually honors.
  • The runner itself stays !Send (Rc internals) — runs execute on the loop thread; the channel is the entire cross-thread input surface.

Superseded-run observability ​

When latest-wins displaces a pending job — or a fifo-bounded key evicts its oldest pending entry — the displaced trigger does not vanish silently. RunSupervisor keeps the displaced Job per supersede key and injects a marker event into the next replacement run for that key, alongside its trigger:

  • Kind push.superseded, source __runner__, payload carrying the displaced trigger's repo/sha (where the event payload has them), the key, superseded: true, and a superseded_by object naming the replacement event's kind.
  • Multiple displacements for one key accumulate in order and all land on the same replacement run; other keys never see them.
  • Because the marker is an ordinary injected event, it is logged in the run's event log and persisted to --history JSONL like any other event — the displaced SHA is auditable after the fact.

No GitHub commit status is posted automatically: status posting is a campaign-level concern (peregrin-sink-github-status needs campaign node execution context, which the queue layer doesn't have). A campaign that wants a "superseded" status on the displaced SHA adds a github-status node matching the push.superseded event type.

Durable queue (--queue-store) ​

Without a store the queue is memory-only — a restart loses every pending trigger. campaign run --queue-store <path> (supervisor mode only) opens a sqlite QueueStore (peregrin-queue-sqlite) and makes the queue durable:

  • What persists — pending submissions grouped by supersede key (per-key writes, so peers never clobber each other) plus the in-flight marker. Written through on every submit/drain/finish.
  • What replays — on boot, RunSupervisor::with_store loads the snapshot and runs it through Queue::restore before the loop accepts new triggers: pending jobs restore in drain order and the interrupted run's marker re-queues at the front (at-least-once — it gets another shot). Under latest-wins a newer same-key pending entry still supersedes it, exactly as live.
  • Re-validation — persisted rows aren't a trust boundary: each entry's stored key must match policy.key_of(event) (a changed supersede: block or corrupt row drops it, logged), and restored entries go through the same dedupe/bound path as live submits, so a restored queue can't hold a state submits couldn't produce.
  • Failure policy — an unreadable store fails the boot (queued triggers would be silently lost otherwise); per-entry corruption is skipped with a warning; write failures mid-run are logged and degrade durability rather than killing runs.

The store file is ordinary sqlite — queue_pending(key, seq, job) rows plus a singleton queue_running row; safe to inspect with sqlite3.

Under peregrin campaign serve --queue-store-dir <dir> each hosted campaign gets its own store at <dir>/<campaign>.db — same schema, same semantics, independent namespaces. Likewise --history-dir namespaces run history as <dir>/<campaign>.jsonl. See Serve a fleet of campaigns.