Fencing Tokens and Drain Protocols for Agent Session Pools

Executive Summary

Retiring a worker, leader, or long-lived connection raises two separate questions: how does the system prevent a stale owner from acting, and how does it know that admitted work has reached a safe stopping point? Production systems answer those questions with several related, but not interchangeable, mechanisms. Kafka transactional producers use ordered epochs, while HDFS JournalNodes enforce single-writer access to the shared edit log. Kubernetes, load balancers, and application servers separate intake control from completion. Stream processors expose progress estimates through watermarks. The reusable lesson is to combine resource-enforced fencing, staged drain, and observable completion evidence without pretending that one mechanism proves the guarantees of another.

AI agent runtimes are now hitting these same problems from a different angle. A pool of long-lived agent sessions — each holding conversation state, a prompt cache, maybe an open tool-use loop — is architecturally close to a pool of long-lived TCP connections or Temporal workflow executions. When you need to retire a session, migrate it, or cut traffic over to a new topology, you inherit every one of the classic failure modes: split-brain (two processes both think they own a channel), lost work (killing a process mid-task), and false completeness (believing a drain finished when a straggler request was still in flight). This piece surveys fencing tokens, drain protocols, watermarks, and explicit policies for ambiguous in-flight work, then distills the transferable design rules.

Fencing Tokens: The Core Primitive

The clearest statement of the problem is Martin Kleppmann's 2016 critique of Redlock, How to do distributed locking. Kleppmann's argument isn't really about Redis — it's about what a lock is for. A lock is only useful if it prevents two clients from concurrently acting on a shared resource, and no lease-based lock can guarantee that on its own: a client can acquire a lock, then experience a GC pause or network delay long enough for its lease to expire, then resume and write to the resource believing it still holds the lock — after a second client has already acquired the lock and started its own work.

The fix is not a better lock, it's a fencing token: "a fencing token is simply a number that increases (e.g. incremented by the lock service) every time a client acquires the lock." The token by itself does nothing — the enforcement has to happen at the resource being protected. The storage service must "remember that it has already processed a write with a higher token number" and reject any write carrying a lower token. This is the crucial and often-missed part of the pattern: the lock service issuing tokens and the resource enforcing them are two different components, and the safety property lives entirely in the second one. Kleppmann's specific complaint about Redlock is structural: "Redlock does not have any facility for generating fencing tokens" — its lock value is a random string with no ordering guarantee, so even a theoretically perfect Redlock implementation gives you mutual exclusion under good conditions but no protection against a resumed zombie client.

This pattern — an ordered value checked at the point where a stale action could take effect — appears directly in some systems and only by analogy in others:

ZooKeeper zxid — ordered history, not an external-resource fencing token. A zxid contains a leader epoch and a per-epoch counter, giving ZooKeeper proposals a total order and helping recovery select a sufficiently up-to-date leader. That makes zxid useful as an analogy for epochs, but ZooKeeper does not thereby cause an arbitrary downstream database or tool to reject a stale client's write. That last enforcement step is what makes a value a fencing token for the protected resource. (ZooKeeper Internals)
Kafka transactional producer epochs. KIP-98 introduced a producer id and epoch so a new instance using the same transactional.id can fence an older producer generation and recover or abort its unfinished transaction. KIP-447 later changed how producer identities are managed for scalable exactly-once processing; it did not originate the fencing mechanism. Brokers and the transaction coordinator perform the decisive checks. (KIP-98; KIP-447)
Kafka consumer-group generations and member epochs. The classic group protocol uses a generation to reject operations from superseded members. KIP-848 moves the new consumer protocol to coordinator-managed member epochs and defines reconciliation around those epochs. This serves the same broad purpose — rejecting stale group state — but it is a distinct protocol from transactional-producer fencing. (KIP-848)
HDFS NameNode fencing with QJM. In Quorum Journal Manager mode, JournalNodes permit only one NameNode writer at a time, protecting the shared edit log from split-brain writes. Hadoop still recommends configuring an external fencing method because an old Active can continue serving stale reads until it shuts down after a rejected journal write. QJM's journal-writer exclusion and process-level fencing therefore cover different failure surfaces; neither is merely a backup for the other. (HDFS High Availability with QJM)

The common shape of the true fencing cases is narrower: an authority advances an ordered generation, and the component capable of accepting the stale operation checks that generation before applying it. ZooKeeper's zxid shows how epoch ordering supports consensus history, but it should not be counted as proof that an unrelated resource is fenced. Losing the downstream comparison — accepting any token, or accepting based on wall-clock recency instead of the token value — reopens the race Kleppmann described.

Drain and Quiesce in Production Infrastructure

Fencing answers "how do we stop a stale actor from acting." Draining answers a related but distinct question: "how do we stop routing new work to an actor while giving its existing work time to finish." Three well-documented implementations show the shape of the pattern.

Kubernetes node drain. kubectl drain first marks the node unschedulable, then evicts eligible pods. Cordoning controls future scheduling; it does not stop the pods already there. Evictions normally respect PodDisruptionBudgets, and kubectl drain retries when the API rejects an eviction. An operator can explicitly bypass that protection with --disable-eviction, which changes the safety boundary rather than proving the disruption is safe. During pod termination, a preStop hook runs before the TERM signal, but the hook and normal container shutdown share the pod's termination grace period — the hook does not receive an additional independent window. A successful drain therefore establishes that the selected pods were removed through the requested policy, not that application-level requests inside them all completed. (Safely Drain a Node; Container Lifecycle Hooks)

Load balancer connection draining. For an Application Load Balancer target group, AWS calls the mechanism deregistration delay. A deregistering target enters draining; the load balancer stops routing new requests to it while allowing in-flight requests time to complete. The interval is configurable from 0 to 3,600 seconds, with a 300-second default. If the target closes its connection before that interval elapses, clients can receive 5xx responses, so application shutdown and load-balancer timing must be coordinated. Health-check failure is a separate transition, and Network Load Balancers expose additional connection-termination behavior, so neither should be folded into the ALB deregistration contract. (ALB target-group attributes; NLB target-group attributes) Envoy exposes a related but not identical control: /healthcheck/fail can make the process fail health checks, while graceful draining discourages new traffic and gradually closes connections over the configured drain interval. (Envoy draining)

Application-level graceful shutdown. Go's http.Server.Shutdown() closes listeners and idle connections, then waits indefinitely for active connections to become idle unless the supplied context expires; it does not wait for or close hijacked connections such as WebSockets. nginx asks old workers to shut down gracefully on reload, and worker_shutdown_timeout can bound that graceful period. These are process-local completion mechanisms, not proof that an upstream router has already stopped intake, so production shutdown still has to coordinate both sides. (Go Server.Shutdown; nginx worker_shutdown_timeout)

The pattern across the examples is: draining separates intake control from a controlled wait for existing work, often with a deadline or escalation policy. Cordon precedes eviction; target deregistration precedes process exit; listener closure precedes waiting for handlers. Keeping those responsibilities distinct prevents a router or scheduler from admitting fresh work solely because the worker has begun local shutdown.

Two-Phase Shutdown: Why "Stop Intake" Needs Its Own Acknowledged Handshake

A subtlety that's easy to underweight in distributed routing is that "stop accepting new work" may not be a fact the shutting-down component can assert unilaterally. Cordoning a Kubernetes node changes scheduler state rather than stopping pods. A process that fails its health check still depends on the upstream traffic selector observing and honoring that state. Whenever the component that wants to stop differs from the authority that controls admission, the protocol needs observable confirmation from that authority rather than trust in a local flag.

The Chandy-Lamport distributed snapshot algorithm offers a useful analogy, with an important scope boundary. Its marker messages record a consistent global state over reliable FIFO channels; it is not a shutdown or drain protocol. What transfers is the reasoning discipline: a claim about a distributed boundary must account for messages in transit and for every participating channel, rather than relying on one process's local flag. (Chandy and Lamport, Distributed Snapshots)

Epoch-based reclamation (EBR) in lock-free concurrent programming is a narrower but structurally similar analogy. EBR lets threads defer freeing memory that other threads might still be reading, by tagging retired objects with the current global epoch and only reclaiming objects retired at least two epochs in the past — guaranteeing every thread has "quiesced" through the intervening epoch before reclamation happens. (Lock-freedom without garbage collection) The transferable idea isn't the memory-reclamation mechanics, it's the shape: an epoch boundary only becomes safe to act on once every participant has provably crossed it, and that's established by an epoch counter everyone advances through together, not by a timeout.

For an agent-session pool, a staged fence can therefore be a sensible design choice: first reject new conversation turns, then confirm the intake authority has recorded that state, then widen the restriction if the deadline expires. The exact stages and any compare-and-swap acknowledgment are proposed protocol requirements for that system; Kubernetes and Envoy supply analogous ordering, not a universal specification for those stages.

Watermarks: "Complete" Is a Frontier, Not a Counter

Stream processing systems solve an adjacent problem: not "has this component stopped," but "how far does the system believe event time has progressed?" Flink computes an operator watermark from the minimum of its active input watermarks so one lagging partition holds the frontier back. Beam is explicit that its watermark is an estimate of input completeness, not proof that no earlier event can ever arrive. Late-data policies exist precisely because the estimate can be wrong. (Flink time and watermarks; Basics of the Beam model)

The distinction still matters for drain protocols: "highest id observed" is not evidence of gap-free completion. If a session pool requires the exact guarantee that no accepted item at or below boundary N remains unfinished, it needs an authoritative admission/completion ledger, per-item acknowledgments, or another mechanism that can prove the set is empty. Calling that exact boundary a watermark borrows useful vocabulary but overstates what ordinary event-time watermarks guarantee. The transferable rule is to define the required frontier precisely and expose evidence matched to that claim.

Ambiguous In-Flight Work Needs an Explicit Policy

Every drain protocol eventually has to answer what happens when its deadline expires while work is still running or its status is unknown. The examples above do not establish one universal fail-closed policy: a Kubernetes eviction can remain blocked by a disruption budget or be bypassed by an explicit operator flag; an ALB ends its deregistration wait without creating an application-level transaction record; Kafka can reject a fenced producer and resolve an old transaction because its coordinator owns that protocol state.

For an agent-session pool, the safe policy must therefore be designed rather than inferred. Where duplicate side effects or split-brain ownership are unacceptable, a strong default is to reject stale-epoch actions and record each unresolved item as expired, aborted, or needs_reconciliation. Where work is safely idempotent, controlled retry or adoption may be valid. In either case, silence is the dangerous outcome: the ledger should preserve what was admitted, which epoch owned it, what terminal decision was made, and whether human or automated reconciliation is still required.

Mapping to AI Agent Session Pools

A pool of long-lived AI agent sessions shares the structural properties that make all of the above necessary: sessions are long-lived (not request-scoped), they hold state that's expensive to lose (conversation history, tool-call context, cached prompts), and — increasingly — they're managed by an orchestration layer distinct from the process that actually receives new messages for them, which is exactly the "who controls the claim path" problem from the two-phase section above.

Two agent-orchestration frameworks illustrate how these concerns can be split into narrower mechanisms:

Temporal Worker Versioning routes new and Auto-Upgrade workflows according to the Current/Ramping deployment configuration, while Pinned workflows continue to generate workflow tasks for their pinned version. Temporal's drainage status is derived from periodic counts of open pinned workflows; it is useful operational evidence, but not an instantaneous, gap-free proof. Decommissioning therefore combines routing state, observed drainage, and operator policy rather than treating process uptime as sufficient evidence. (Temporal Worker Versioning; Sunset and GC source)
LangGraph durable execution persists graph state at checkpoint boundaries so an interrupted workflow can resume from recorded state. It does not mean an arbitrary process can be killed after every superstep with all side effects automatically safe: applications must use a checkpointer and make side effects deterministic or idempotent, commonly by isolating them in tasks. This is a checkpoint-and-resume option, not a complete drain protocol by itself. (LangGraph durable execution; LangGraph persistence source; Re-execution and idempotency source)

Both provide useful building blocks without collapsing into the same mechanism. Temporal separates version-aware routing from an eventually refreshed drainage observation. LangGraph separates durable checkpoints from the application's responsibility for replay-safe side effects. Neither substitutes for a resource-enforced fencing token when two live processes can still reach the same external side effect.

Transferable Design Rules

A token is only as good as the check on the other end. Issuing a monotonically increasing generation/epoch number accomplishes nothing unless the component that actually performs the sensitive action — the write, the delivery, the process kill — compares the incoming token against the highest it has already accepted and rejects anything lower. The token-issuing service and the token-enforcing resource are different responsibilities; conflating them (or skipping the comparison) is exactly the gap that broke Redlock.
Separate "stop intake" from "wait for existing work," and require observable confirmation from whoever controls admission. Cordon before evict, deregister before process exit, listener-close before handler-wait. For an agent pool, a versioned compare-and-swap acknowledgment is one concrete design that can make the admission transition auditable; it is a recommendation here, not a mechanism supplied by every cited system.
For agent pools, prefer staged scope escalation when different traffic classes carry different risks. Stop the cheapest, most reversible intake class first, confirm it took effect, then widen the fence only if policy requires it. This is an explicit design recommendation; the cited infrastructure examples motivate the ordering but do not prescribe these agent-specific stages.
Exact completeness requires stronger evidence than a max-seen counter or event-time watermark. "Nothing admitted at or below boundary N is outstanding" must come from a gap-aware ledger or acknowledgments that can prove that set empty. A highest dispatched id, or a heuristic event-time watermark, cannot establish that claim alone.
Make the timeout policy explicit and auditable. For non-idempotent side effects, reject stale epochs and record unresolved work for expiry, abort, or reconciliation. For safely idempotent work, retry or adoption may be acceptable. The invariant is that the system never silently loses ownership history or treats an unknown outcome as success.
Give the protocol an authoritative admission/completion ledger and an indexed drain query. Kubernetes does not expose a universal "eviction status" object, and Temporal's drainage status is itself periodically computed from workflow counts. An agent pool that needs stronger guarantees should define its own observable conditions — for example, no accepted-but-nonterminal items for the fenced epoch — and make that query efficient enough to run during every drain.

None of these ingredients is new. What is newer is their combination in agent-session pools: long-lived state, externally visible side effects, and an orchestration layer that may reshape topology while work is active. The prior art supplies tested primitives and sharp boundary conditions; the session-pool protocol still has to state exactly which resource enforces the fence, who owns admission, and what evidence is strong enough to declare the drain complete.