Skip to content

Multi-Machine Session Pool

When you run the same agent on more than one machine — a laptop and a Mac mini, say — Instar can treat them as a single pool of compute rather than a primary with a cold standby. Every machine is awake; each conversation is placed on the best-fit machine; and you can move a live conversation from one machine to another mid-stream just by naming it (“move this to the mini”). The conversation continues coherently on the new machine, and no message is ever dropped or doubled.

This page maps the layers that make that work. The whole subsystem ships dark (off until you explicitly enable it through a staged rollout), and a single-machine agent behaves exactly as it does today.

L0 — Machine-to-machine backbone (MeshRpc)

Section titled “L0 — Machine-to-machine backbone (MeshRpc)”

Machines talk over a thin, signed request/response layer. MeshRpc carries a small command set (place, claim, release, transfer, deliverMessage, a2a-inbox-deliver, capacity-report, secret-share) inside an envelope that is Ed25519-signed, bound to a specific recipient, and protected against replay by a nonce + timestamp. The receive side (MeshRpcDispatcher) runs a five-step verification — recipient, signature, registered-peer, unseen-nonce, fresh-timestamp — and a per-command authorization gate before any handler runs; a rejected command never even burns its nonce. For agent-to-agent delivery, A2aMeshInbox additionally binds the authenticated sender machine to the configured agent principal and requires the strict marker principals to match before dispatching into the recipient’s existing inbox allowlist. The send side, MeshRpcClient, builds and signs the envelope, POSTs it to a peer’s /mesh/rpc, and surfaces the typed result. Together MeshRpcClient and the dispatcher are the secure transport every other layer rides on.

Exactly one machine holds the router role at a time, decided by a fenced lease (the same monotonic-self-fence discipline the cross-machine lease already uses). The router is the single writer of placement decisions in steady state; if it goes silent, another machine re-elects and resumes.

LocalLeaseStore provides the local lease record when git-backed lease storage is unavailable. A local record alone cannot prove a peer has yielded, so the coordinator treats contested ownership as a conflict instead of using the local tie-break as permission to run two routers.

L2 — Machine pool registry (MachinePoolRegistry, NicknameAssigner)

Section titled “L2 — Machine pool registry (MachinePoolRegistry, NicknameAssigner)”

The MachinePoolRegistry tracks every machine’s live capacity — load average, active session count, memory pressure, capabilities, hardware — plus a clock-skew quarantine state machine that pulls a machine with a bad clock out of placement. Each machine gets a friendly, editable nickname (auto-assigned by NicknameAssigner, deduplicated against the pool) so you never have to type a raw machine id. The registry and NicknameAssigner together back the dashboard’s Machines tab and the nickname you use in a transfer command.

L3 — Per-session ownership (SessionOwnership, SessionOwnershipRegistry)

Section titled “L3 — Per-session ownership (SessionOwnership, SessionOwnershipRegistry)”

Ownership of a conversation is movable and fenced by a (status, epoch) pair. SessionOwnership is the pure state machine — place → claim → transfer → release, with the epoch arithmetic and legal sequencing enforced in one place. SessionOwnershipRegistry sits on top and performs the compare-and-set: a single-ref fast-forward push decides the winner when two machines contend, never a machineId comparison. The exactly-one-owner invariant is structural — a worker runs a session only while it observes itself as the active owner at the current epoch, and the SessionOwnershipRegistry rejects a stale or replayed claim.

L4 — Placement + routing (PlacementExecutor, NicknameCommand, SessionRouter, DeliverMessageHandler)

Section titled “L4 — Placement + routing (PlacementExecutor, NicknameCommand, SessionRouter, DeliverMessageHandler)”

PlacementExecutor is the single, canonical placement brain. Its policy is structured data (weights, thresholds, ordering), validated against a fixed schema at startup, so every machine in a pool makes consistent decisions. PlacementExecutor.decide() is pure and ordered — hard-constraint, then pin, then sticky, then least-loaded — and on an unsatisfiable hard pin or capability it queues and escalates rather than silently mis-placing.

For each inbound message the SessionRouter resolves ownership and dispatches: handle locally if it owns the session, forward over MeshRpc to the owner otherwise, re-place on owner death, or — for a brand-new conversation — call PlacementExecutor, claim ownership synchronously, and spawn. The owner side runs DeliverMessageHandler, which records each forwarded message in the processing ledger before acting on it; a redelivered message is ACKed as a duplicate and never re-processed, so a handoff can’t double-reply. The SessionRouter dispatches strictly in order per session and advances the platform offset only after DeliverMessageHandler confirms durable receipt.

Self-placement confirmation stays outside the router: SessionPoolLocalClaim advances a self-owned placing row only after the established local inject or spawn tail succeeds. Active, remote-owned, and missing rows are no-ops; a failed local delivery never confirms ownership, while a later observer failure does not reverse a compare-and-set that already committed.

When you say “move this to the mini” or “run this on the workstation”, the NicknameCommand recognizer parses that — conservatively, only on an explicit relocation verb plus a known nickname, so a passing mention of a machine never triggers a move. NicknameCommand resolves the nickname against the registry; an unknown name is rejected with the valid options rather than mis-routed.

L5 — Transfer / handoff (TransferByNickname, TransferOrchestrator)

Section titled “L5 — Transfer / handoff (TransferByNickname, TransferOrchestrator)”

TransferByNickname turns a recognized relocation request into a gated plan: it resolves the target, enforces a per-topic rate limit, no-ops if you’re already there, and requires confirmation when the target is offline or you’re mid-reply. TransferByNickname is the planner; TransferOrchestrator executes the move. TransferOrchestrator drives the ordered handoff — active(source) → transferring → active(target) → source-release — which is claim-before-release: because the fence is the status+epoch (not the physical release), there is never a double-run and never a no-owner gap. It bounds the drain of an in-flight reply, enforces an output-exclusion window so the two machines’ output never interleaves, and verifies the ledger snapshot is fully settled before the target resumes — if the synced state is incomplete, TransferOrchestrator escalates honestly instead of resuming from corrupted context.

L6 — Multiple agents per machine (MachineLoadBroker)

Section titled “L6 — Multiple agents per machine (MachineLoadBroker)”

When several agents share one machine, each agent’s router must see the true machine load, not just its own. MachineLoadBroker accounts for every resident agent’s sessions and — crucially — does not trust agent self-reports for the authoritative number: it cross-checks each agent’s claimed footprint against the OS-measured footprint. An agent that reports itself idle while actually saturating the box is flagged suspect-overloaded, so a lying agent can never attract sessions onto an already-busy machine. MachineLoadBroker also verifies the isolation invariant — distinct ports, identities, and non-nested home directories.

Size-aware placement and heavy work (PlacementExecutor, JobLoader, JobScheduler, LocalVerifyService)

Section titled “Size-aware placement and heavy work (PlacementExecutor, JobLoader, JobScheduler, LocalVerifyService)”

Load is compared per core, so a big machine and a small one at the same raw load are not treated as equally busy: PlacementExecutor divides each machine’s load by its core count before picking the least-loaded one. Work also has a weight — light, standard or heavy. A machine counts as big enough for heavy work when it has at least 12 cores and 32 GB of memory (fixed in code); a machine whose size is unknown never counts. Each machine can override this with its own heavyWork setting: allow makes it eligible regardless of size, deny keeps heavy work off it. Among eligible machines, heavy work goes only to one with room right now; when none has room it is not piled onto a full machine (each path below says what happens instead).

  • Jobs. A job can declare workWeight (light, standard or heavy); JobLoader refuses any other value. JobScheduler skips a heavy job on a machine that is not eligible (below the floor, size unknown, or set to deny) and records why — but only while another online machine is eligible, so a machine on its own still runs its heavy jobs. A job whose machines list names this machine always runs here.
  • Test runs. When LocalVerifyService hands a full verify to another machine, it drops peers that are too small, too busy or unmeasurable and tries the rest least-loaded first; when every peer is dropped, the verify runs in this machine’s own slot. Repeated skips of the same peer for the same reason are recorded at most once every ten minutes.
  • Helper work. GET /pool/placement/preview answers “where would this heavy work go?” without changing anything, and POST /pool/heavy-work/spawn starts a background helper session on the machine the preview picks, or refuses with a reason when no machine fits.

Two neighbouring readers stay out of the decision. MachineLoadAssessor is an advisory, shadow-first check: when a machine’s own load verdict and the outside view of it disagree, a fast model labels the snapshot and grades that label fifteen minutes later, but nothing in placement reads its output. PlacementResponderReader answers the opposite question after the fact — which machine actually served a reply — by reading the topic’s current owner from GET /pool/placement, so live cross-machine proofs can check that work landed where placement said it would.

Spec: docs/specs/resource-aware-placement.md.

Rollout and rebalance (StageAdvancer, SessionPoolE2EResultStore, RebalancePlanner)

Section titled “Rollout and rebalance (StageAdvancer, SessionPoolE2EResultStore, RebalancePlanner)”

The pool activates through a graduated ladder: dark (code shipped, always-local) → shadow (real placement + ownership, no moves) → live-transfer (failover + explicit pins) → rebalance (load-driven moves). The ladder is enforced in code, not by willpower. SessionPoolE2EResultStore is a signed, append-only record of each stage’s end-to-end test outcome; StageAdvancer is the sole writer of the rollout stage and refuses to advance a stage unless the prior stage’s result is green for the current commit — and StageAdvancer mechanically reverts if a live stage later regresses. A direct config write to the stage is rejected; only StageAdvancer holds the capability to change it.

Promotion activation is separately fail-closed. multiMachine.sessionPool.promotionModel defaults to off; operator enables authenticated one-step requests through POST /session-pool/promote, while auto-climb periodically requests the same one checked step. Both live models reuse StageAdvancer and its signed green evidence, and neither can cross multiMachine.sessionPool.promotionCeiling. Invalid model or ceiling values fall back to the dark posture.

Once at the rebalance stage, RebalancePlanner proposes bounded moves off an over-saturated machine — only non-pinned, low-priority sessions that are off their transfer cool-down, at most one move per source per cycle so it can never cascade. RebalancePlanner is evaluated only on the heartbeat interval, never per message, so a single message can’t trigger a storm of transfers.

The pool doesn’t reinvent the cross-machine primitives Instar already had. The router lease builds on LeaseCoordinator and the existing MultiMachineCoordinator; peer authenticity reuses MachineIdentity (Ed25519 signing + the registered-peer set); the exactly-once guarantee leans on the same MessageProcessingLedger that direct ingress uses; and a transferred session resumes through the existing TopicResumeMap. MachineIdentity and MessageProcessingLedger in particular are load-bearing here — the first is what makes a MeshRpc envelope verifiable, the second is what makes a DeliverMessageHandler redelivery a no-op. Reusing LeaseCoordinator, MultiMachineCoordinator, and TopicResumeMap keeps the pool consistent with the single-machine behavior you already rely on.

  • Exactly-once on channel messages and replies — never dropped, never doubled — across placement, transfer, and failover. (External tool side effects are best-effort-once; carry your own idempotency key for those.)
  • Backward compatible — a one-machine agent is byte-identical to today, and the whole pool stays dark until you advance the rollout stage.
  • Observable — GET /pool shows the router, every machine’s nickname/hardware/load/clock state; GET /session-pool/e2e-results shows the rollout gate’s state; GET /session-pool/failover-runner shows the in-agent failover-drill runner’s status — the dark/dev-gated, dry-run-first background helper that runs the real two-node failover E2E on a live agent and records an honest green/red into the rollout gate’s result store (503 when the runner is dark).

Placement observability and reliable transfer

Section titled “Placement observability and reliable transfer”

Two robustness surfaces sit on top of the L3/L4 machinery so an agent never has to guess where a topic runs or hope a phrase is recognized.

Knowing which machine, and why. GET /pool/placement?topic=N answers the question a standby session previously could not: it returns the owning machine and nickname, the lease-holder, and — critically — the reason a topic is where it is: pinned (a deliberate move, recorded in TopicPlacementPinStore), placed (the SessionRouter/MachinePoolRegistry load-balanced it there — not a deliberate move), or unowned. It is answerable from any machine: a non-holder transparently proxies to the lease-holder, whose TopicPlacementPinStore is authoritative. The pure TopicPlacementDescription helper computes the answer from already-resolved ownership + pin state, so the route stays thin and the logic is unit-tested. This closes a real failure mode where an agent inferred “the move worked” from the fact that it happened to be running on a machine — running on a machine is not the same as having been deliberately moved there.

Reliable transfer, independent of phrasing. The natural-language path (“move this to the mini”) flows through NicknameCommand → TransferByNickname. The recognizer’s known-nickname set is built by RelocationNicknameSet, which unions the local machine’s own nickname (resolved from the registry, the identity entry, or a deterministic derive) into the set — without this, “move it back to the machine currently handling the topic” silently fell through, because the lease-holder runs the relocation check and its own nickname could be missing from MachinePoolRegistry.getCapacities(). For callers that want determinism rather than recognition, POST /pool/transfer (body {topic, to}, where to is a nickname or machineId) runs the same validated TransferByNickname planner — rate-limit, online, already-there, and offline-confirmation gates included — sets the pin, and releases local ownership; a non-holder proxies to the holder. So a transfer never depends on getting the wording right, and an agent (or operator) can move a topic with a single deterministic call.

Both surfaces are read-models/levers over existing state — SessionOwnershipRegistry for ownership, TopicPlacementPinStore for pins, MachinePoolRegistry for nicknames/liveness — and 503 when the pool is dark, exactly like GET /pool and POST /mesh/rpc.