Keeping Watchdogs From Fighting Planned Transitions
Executive Summary
Every self-healing system faces the same trap: the mechanism that keeps it alive during real failures is also the mechanism most likely to sabotage a planned change. A liveness probe that correctly kills a wedged process will, with equal enthusiasm, kill a process that is deliberately taking its time to drain connections and flush state. A watchdog timer that correctly reboots a hung daemon will, with equal enthusiasm, reboot one that an operator just told to stop. The failure is not that the health check lied — it detected exactly what it was built to detect, an absence of liveness signal. The failure is that detecting absence and detecting fault were never the same thing, and most supervisors conflate them.
This article surveys how mature infrastructure systems — Kubernetes, systemd, Erlang/OTP, Nomad, Consul, AWS Auto Scaling, Prometheus Alertmanager, PagerDuty, and the Patroni/Orchestrator database-failover tooling — draw the line between "not responding because it's broken" and "not responding because it's supposed to be doing something else right now." It then extracts the recurring design patterns (single lifecycle owner, explicit intentional-vs-failed state, checking state at action time rather than schedule time, queuing instead of rejecting), adds a proposed generation fence for in-flight work, and applies them to a concrete and under-discussed case: supervisors that keep persistent LLM agent sessions alive via heartbeats, and must not let that supervision fight a planned migration, mode switch, or maintenance window.
The central finding, repeated across every system surveyed, is that disabling new monitoring is not enough. Work that was already in flight before a transition began — a probe scheduled a few seconds earlier, a timer already armed, a health check already dispatched — must also be fenced off, and the fence has to be checked at the moment the remediation action is about to happen, not only at the moment the check was scheduled. Kubernetes' own change history — a deletion guard added in one release, extended to node shutdown in another, and a still-unexplained report of liveness failures during termination — shows how many places such a fence has to reach, even when the intent is documented in the code.
The Core Problem: Health Signals vs. Lifecycle Intent
A health signal answers one question: "did I hear from you recently?" It cannot, by itself, distinguish between:
- a process that is dead or wedged (a fault — remediate)
- a process that is intentionally not responding because it is mid-shutdown, mid-drain, or deliberately paused (not a fault — leave alone)
- a process that will resume responding once a planned operation completes (temporary, not a fault — wait)
Systems that only reason about signal absence will act on all three identically. Systems that survive planned operations gracefully all add a second axis: an explicit, queryable record of intent — "this thing is stopped by design" — that the remediation path must consult before it acts, not just before it schedules a check.
Prior Art Survey
Kubernetes: Probes, Termination, and Disruption Budgets
Kubernetes separates liveness, readiness, and startup probes by the action each one triggers, not just by what it measures. Per the official probes documentation: a startup probe failure causes the kubelet to kill the container and apply the restart policy; a liveness probe failure restarts the container; a readiness probe failure only removes the pod's IP from the EndpointSlice — it never kills anything. This is a deliberate decoupling of "should traffic route here" from "should this process be destroyed," and it means the cheapest, most reversible signal (readiness) is the one allowed to fire constantly, while the destructive signal (liveness) is deliberately harder to trigger and explicitly documented as not waiting on readiness.
The more interesting history is in how the kubelet treats probes during termination, because it shows Kubernetes drawing the line per probe type, by the action each probe controls — and the failure mode it guards against is exactly the one this article is about. kubernetes/kubernetes#98571 ("Stop probing a pod during graceful shutdown," v1.21) added a guard to the kubelet's prober worker: when a pod's DeletionTimestamp is set, liveness and startup probe results are forced to "success" and probing stops, logged as Pod deletion requested, setting %v probe result to success (with the probe type filled in). Separately, kubernetes/kubernetes#105215 (v1.23) addressed node graceful shutdown, which "does not terminate probes upon kubelet shutdown. This can potentially cause a pod to be terminated from a Liveness probe failing."
The guard was not airtight in practice. kubernetes/kubernetes#107473 reports users on EKS and Minikube (versions 1.20–1.22) seeing liveness-probe failure events during the termination window, with the "Pod deletion requested" log line never appearing. What the report documents is failed observations: it does not show that those failures killed or restarted a container early. Contributors in the thread proposed that the probe worker was evaluating a copy of the pod that did not yet carry the deletion timestamp; that explanation was disputed, a later comment noted that "The contract is that a liveness probe and readiness probe can run during shutdown" and that a liveness probe may deliberately set a shorter termination period than the pod's grace period, and the proposed fix (#107514) was closed unmerged. The issue was eventually closed as not planned by the triage bot with the cause unresolved. A stale view of the pod is one possible explanation, not an established one.
Readiness is deliberately different. kubernetes/kubernetes#122824 asked why readiness probes were not included in the same DeletionTimestamp check, since they kept running (and logging failures) during a long preStop window. A maintainer closed it as expected behavior: "Readiness probes during termination are expected. Readiness probes during termination allow for statuses to be propagated to allow for Services to be cleanly removed from Endpoints." The PR linked in that comment, kubernetes/kubernetes#110191, had re-enabled readiness probing on termination after a 2021 kubelet refactor disabled all probes on pod deletion (release note: "Fix 1.22 regression so pods will now post their readiness during termination"), reasoning that "Readiness probes are important for controllers and load balancers to know when to take a pod out of service." The same change stops only liveness and startup probes when a pod enters termination (probeManager.StopLivenessAndStartup(pod)). So Kubernetes does not suppress every probe during a planned stop: it stops the probes whose action is destructive (kill/restart) and keeps the one whose action is to steer traffic, because during a drain that signal is what lets endpoints and load balancers stop routing to the pod. The design lesson is to scope suppression to the action a check controls, not to the check itself.
Above the single-pod level, PodDisruptionBudgets apply the planned/unplanned distinction to eviction admission: a PDB limits how many pods of a replicated application voluntary evictions through the Eviction API may take down at once. It is not a rollout control. Per the disruptions documentation, pods deleted or unavailable during a rolling upgrade count against the budget, but "workload resources (such as Deployment and StatefulSet) are not limited by PDBs when doing rolling upgrades"; rollout availability is configured in the workload's own spec. Deleting deployments or pods directly also bypasses PDBs. Involuntary disruptions (node hardware failure, kernel panic) "cannot be prevented by PDBs; however they do count against the budget." kubectl drain, per the safely drain a node guide, evicts pods through the Eviction API, respecting PDBs and graceful termination periods — unlike directly deleting pods, which the disruptions documentation advises cluster tooling against for exactly that reason. kubectl cordon only marks a node unschedulable for new work; it is a separate, reversible step from actually removing existing work via drain. The two-step cordon-then-drain sequence is itself a lifecycle-ownership pattern: "stop admitting new work" and "remove existing work" are deliberately split, so that the scheduler stops placing ordinary new pods on the node (pods that tolerate the unschedulable taint are the documented exception) while existing ones are moved off through the budget-respecting eviction path.
systemd: Watchdog Timers and the Stop-Job Exemption
systemd's software watchdog (WatchdogSec=) requires the service to call sd_notify(WATCHDOG=1) at roughly half the configured interval; miss enough pings and systemd marks the unit failed and sends SIGABRT (systemd.service(5), sd_notify(3)). Critically, the restart policy (Restart=always, on-failure, etc.) is explicitly suspended for operator-initiated stops: systemd will not restart a unit if it was stopped via systemctl stop or an equivalent operation, or if its exit status matches RestartPreventExitStatus=. The watchdog and the restart policy both key off of how the unit ended, not merely that it ended — an intentional stop and a crash are different states as far as systemd's remediation logic is concerned, and that distinction is checked at the moment systemd decides whether to restart, not only at the moment the stop was requested.
Erlang/OTP: Supervisors and the Shutdown/Crash Distinction
OTP's supervisor behaviour makes the same distinction a first-class part of the restart contract. Per the Supervisor Behaviour design principles and the supervisor module reference, a child's restart type (permanent, transient, temporary) governs whether it is restarted based on why it exited: a transient child is restarted only on an abnormal exit — any reason other than normal, shutdown, or {shutdown, Term} — while a temporary child is never restarted regardless of exit reason. Calling supervisor:terminate_child/2 to deliberately stop a child does not trigger the supervisor's automatic-restart machinery; the supervisor is the single authority that distinguishes "I told it to stop" from "it died." OTP also bounds remediation itself: intensity/period (by default 1 restart per 5 seconds) caps how many times a supervisor will attempt to resurrect its children before giving up, terminating everything, and escalating failure upward with its own shutdown exit — a deliberate circuit breaker against a restart loop fighting a problem it cannot fix.
Nomad and Consul: Drain and Maintenance Mode
HashiCorp Nomad's node drain marks a node ineligible for new scheduling and migrates its existing allocations according to the job's migrate block. The reschedule block documentation states plainly that "when a node is drained, Nomad migrates the allocations instead and ignores the reschedule block" — a drain-triggered move is explicitly carved out of the failure-triggered reschedule/backoff path, so planned evacuation never looks like, or competes with, failure recovery (see also the node drain command reference and HashiCorp's Advanced Node Draining post).
Consul's maintenance mode works by reusing its own health-check machinery rather than inventing a parallel signal: consul maint -enable -reason "..." registers a synthetic health check in critical status against the service, and -disable deregisters it (consul maint command reference). This durably removes the service from DNS/API "healthy" results — including surviving agent restarts — without requiring every downstream consumer of Consul health data to understand a second, separate "maintenance" concept. The tradeoff is that it is an explicit, durable marker an operator must set and clear; nothing infers it from a missed check.
AWS Auto Scaling: Standby, Suspended Processes, and Lifecycle Hooks
AWS Auto Scaling groups give an operator three distinct levers, all aimed at the same goal — keep an instance alive and off the chopping block while someone works on it:
- Suspended processes:
SuspendProcessescan target theHealthCheckandReplaceUnhealthyprocesses independently ofLaunch/Terminate/others, letting an operator patch or reboot instances without the ASG treating the resulting unresponsiveness as a failure to remediate (Suspend and resume processes, considerations). - Standby state:
EnterStandby/ExitStandbypulls an in-service instance out of load-balancer rotation and out of health-check jeopardy while it remains a group member, specifically so it can be rebooted or inspected without the ASG terminating it (Temporarily remove instances from your Auto Scaling group). - Lifecycle hooks:
autoscaling:EC2_INSTANCE_TERMINATING/LAUNCHINGhold an instance in aTerminating:WaitorPending:Waitstate — default timeout one hour, extendable to 48 hours via heartbeats — so custom drain or bootstrap logic can finish before the ASG's own lifecycle proceeds (Lifecycle hooks overview).
AWS's own documentation also flags the sharp edge version of the "stale state" problem directly: if an instance is terminated outside the normal flow while in Standby, the ASG keeps reporting it healthy until a later health check or re-activation attempt catches up — a documented gap between the state the system believes and the state that is actually true, which is precisely the race this article is concerned with.
Prometheus Alertmanager: Silences vs. Inhibition (Notification Suppression, Not Action Suppression)
Alertmanager offers two mechanisms that are frequently confused: silences, time-boxed, label-matched mutes typically configured for a maintenance window, which expire on their own; and inhibition, a config-defined rule where a firing "source" alert suppresses notifications for matching "target" alerts, encoding "don't page about symptoms when the root cause is already firing" (Alertmanager documentation). Both operate purely at the notification layer — the underlying alert still evaluates, still fires, and still appears in the API and UI. Neither one suppresses an external actuator's remediation: if some other system is watching the same underlying metric and taking action on it directly (not reading Alertmanager's suppressed notification stream), silencing or inhibiting the alert in Alertmanager does nothing to stop that actuator. This is an important cautionary example for anyone tempted to treat "alert went quiet" as equivalent to "nobody will act on this" — alert suppression and action suppression are not the same system, and conflating them is itself a failure mode.
PagerDuty Maintenance Windows
PagerDuty's maintenance windows go further than pure notification muting: during a defined window for a service, PagerDuty does not create incidents at all, not merely suppress their delivery (Event Management documentation). A related but distinct mechanism, alert suppression in Event Orchestration, accepts and stores events for later forensic review without creating an incident from them — "queue, don't discard" rather than "ignore entirely." PagerDuty also separately offers Auto-Pause Incident Notifications, an algorithmic pause for noisy or likely-transient alerts, distinct from an operator-declared maintenance window.
Database Failover Managers: Patroni Pause and Orchestrator Downtime
Patroni's patronictl pause --wait puts a PostgreSQL cluster into maintenance mode: Patroni stops driving automatic failover decisions and will not even restart a stopped Postgres instance on its own, while the cluster's state remains intact in the distributed configuration store until patronictl resume re-enables automation (patronictl documentation). This is explicitly designed for major-version upgrades and other manual interventions where the operator, not Patroni, must be the sole author of what happens next.
Orchestrator (the openark MySQL topology manager) provides begin-downtime, which explicitly marks an instance as downtimed so that automated recovery and failover logic skips it during manual work (topology-recovery.md; implementation in downtime.go). Maintainer discussion in issue #1417 is worth noting for its own sake: begin-downtime and the separate begin-maintenance marker are not interchangeable, and conflating two different "this is planned" signals inside the same tool was itself a source of confusion — a small case study in how even the designers of a fencing mechanism can accumulate more than one incompatible notion of "intentional."
Comparative Summary
| System | Planned-state signal | What it suppresses | Scope | Expires automatically? |
|---|---|---|---|---|
| Kubernetes | DeletionTimestamp on pod; PDB; cordon | Liveness/startup-triggered kill/restart (readiness keeps reporting by design); voluntary eviction beyond PDB budget (not workload rollouts) | Per-pod / per-workload | Yes (deletion completes; PDB is structural) |
| systemd | Unit "stopping/deactivating" state from systemctl stop | Watchdog-triggered restart, policy-based restart | Per-unit | Yes (stop completes) |
| Erlang/OTP | Exit reason normal/shutdown; terminate_child | Supervisor auto-restart | Per-child process | Yes (immediate) |
| Nomad | Node drain / migrate block | Reschedule/backoff logic | Per-node allocation | Yes (drain completes) |
| Consul | consul maint -enable (synthetic critical check) | Healthy status in DNS/API | Per-service | No — manual -disable required |
| AWS Auto Scaling | Standby state; suspended HealthCheck/ReplaceUnhealthy; lifecycle hook wait state | Instance replacement/termination | Per-instance | Lifecycle hooks: yes, bounded timeout; Standby/suspend: no |
| Alertmanager | Silence; inhibition rule | Notification delivery only | Per-alert label match | Silences: yes, time-boxed; inhibition: while source alert fires |
| PagerDuty | Maintenance window | Incident creation and notification | Per-service | Yes, time-boxed |
| Patroni | patronictl pause | Automatic failover and restart decisions | Per-cluster | No — manual resume required |
| Orchestrator | begin-downtime | Automated recovery/failover targeting | Per-instance | Yes, duration-bound |
The pattern that stands out in this table: mechanisms that are not automatically time-boxed (Consul maintenance, Patroni pause) are exactly the ones whose own documentation and community practice warn operators to remember to clear them — a recurring operational hazard discussed further below.
Failure Modes: When Remediation Fights the Plan
Liveness failures during termination (Kubernetes, guarded but still reported). As detailed above, #98571 stopped liveness and startup probing once a pod is marked for deletion and #105215 stopped a pod's probes on node graceful shutdown, a path the first guard had not covered, yet #107473 reported liveness failures during termination on 1.20–1.22 clusters and was closed without a merged fix or an established cause; it records failure events, not a demonstrated early kill. (#122824 is not part of this failure mode: readiness probing during termination is intended.) The lesson is not that Kubernetes is poorly engineered — it is that "check an intentional-stop flag before taking a destructive action" is a pattern that has to be applied everywhere such a decision is made; the node-shutdown path that #105215 had to close separately shows how a single missed call site leaves the exposure open.
Stale health state surviving a state transition (AWS Auto Scaling). AWS's own documentation warns that terminating an instance in Standby outside the normal API flow leaves the ASG believing it is still healthy until a later check or reactivation attempt catches up. This is a textbook case of a cached belief about state outliving the real state change that invalidated it — exactly the race a transition-aware supervisor must close by re-validating state at the moment of action, not trusting a snapshot taken earlier.
Automated failover acting on a brief, maintenance-triggered partition (GitHub, October 2018). GitHub's own post-incident analysis traces the incident to "routine maintenance work to replace failing 100G optical equipment," which cut connectivity between its US East Coast network hub and primary East Coast data center for 43 seconds. During that partition, Orchestrator — which is "built on top of Raft for consensus" and had been active in the East Coast data center — "began a process of leadership deselection"; the Orchestrator nodes in the US West Coast data center and the US East Coast public cloud "were able to establish a quorum and start failing over clusters to direct writes to the US West Coast data center." The East Coast database servers held "a brief period of writes that had not been replicated to the US West Coast facility," and once the application tier began writing to the new West Coast primaries, "the database clusters in both data centers now contained writes that were not present in the other data center," so GitHub could not safely fail back. The result was 24 hours and 11 minutes of degraded service; GitHub stated that "no user data was lost," while "manual reconciliation for a few seconds of database writes" was still in progress when the analysis was published. The postmortem's own diagnosis is that "Orchestrator's actions behaved as configured, despite our application tier being unable to support this topology change," and its first technical follow-up is to "prevent the promotion of database primaries across regional boundaries." The lesson for this article is not about fencing: a short loss of connectivity, itself caused by maintenance, drove an automated remediation that was correct by the failover manager's own rules but could not be safely reversed by the system around it. Remediation authority has to be bounded by what the rest of the system can absorb, not only triggered by the disappearance of a health signal.
Generic failure modes worth naming explicitly, even without a single attributable public postmortem:
- Maintenance-flag races with already-scheduled actions. A probe, timer, or check scheduled a moment before a maintenance flag is set can still be in flight when the flag flips, and if the actuator only consults the flag at scheduling time rather than at the moment it is about to act, the in-flight action proceeds as if nothing had changed. This is the gap the Kubernetes
DeletionTimestampcheck is meant to close for liveness and startup probes — the probe types whose action is destructive. Reading #107473 as this gap being hit in practice is our inference, not something the issue establishes: it documents liveness failures during termination, not an in-flight probe triggering a kill, and its cause was never settled. - Stuck maintenance mode. Mechanisms without automatic expiry (Consul's synthetic critical check, Patroni's pause) rely on an operator to clear them; if that step is forgotten, the system is silently left permanently out of self-healing with no built-in alarm. This is a known enough operational hazard that community documentation for Consul and Patroni both carry explicit reminders to disable/resume after the work is done.
- Unhealthy state reused as a reason to shed or fast-fail traffic. A component marked "unhealthy" for monitoring purposes is sometimes read by an unrelated system (an inbound request router, a load balancer, a circuit breaker) as grounds to reject or drop traffic — even when the underlying cause is a planned, temporary unavailability rather than a fault. Alertmanager's silence/inhibition design (muting notifications without touching the underlying alert or any external actuator reading the same metric) is the clearest illustration of how easily "we muted the alarm" and "we told every consumer this is expected" can diverge.
Design Patterns Distilled
Single owner of lifecycle per phase. Across every system surveyed, exactly one actor is authoritative for a resource's lifecycle during a given phase, and every other actor defers to it. Patroni's pause hands Postgres's fate entirely to the operator; Orchestrator's begin-downtime tells the automated recovery logic this instance is someone else's responsibility; OTP's supervisor is definitionally the only thing allowed to decide whether a child's exit warrants a restart. The corollary for a transition orchestrator: while a transition record is open, it — not the steady-state health engine — is the only actor permitted to act on the lifecycle of what it is transitioning.
Explicit state: intentional vs. failed. systemd's stop-job exemption, OTP's shutdown/normal exit reasons, Nomad's drain-triggered migration being excluded from the reschedule block, and Kubernetes' DeletionTimestamp check all encode the same idea: "ended because it was told to" must be a distinguishable, inspectable state, not something inferred after the fact from the shape of the failure.
Generation/epoch fencing (proposed). This one is a proposal for supervisors rather than a pattern drawn from the incidents above. The motivation is the in-flight-work problem: a probe dispatched before a transition opened can resolve after it, and the action-time check below is only reliable if a late result can be recognized as belonging to an earlier state. Borrowing the monotonically increasing term idea from consensus protocols such as Raft, a transition record should carry a generation number that every in-flight probe or action is tagged with at schedule time, so results tagged with a stale generation can be discarded outright rather than evaluated at all.
Check the transition record at action time, not only at schedule time. This is the single most consequential pattern in this survey. Kubernetes added the deletion guard in the prober worker (v1.21), needed a separate change for node graceful shutdown (v1.23), and later moved probe start/stop into the pod worker's termination path (#110191), while #107473 still reported liveness failures during termination, with no established cause. The practical rule: the moment immediately before any kill/restart/reject action executes must re-read the current transition state, never relying on a flag's value from when the action was first scheduled.
Queue, don't reject. PagerDuty's Event Orchestration alert suppression stores events for later review rather than discarding them; AWS's lifecycle hooks hold an instance in a wait state rather than letting the ASG's normal flow proceed past it. The pattern generalizes: when a resource is intentionally unavailable, consumers of that resource should buffer or defer rather than fail fast, because "unavailable right now" is not the same claim as "unavailable, stop trying."
Bounded maintenance with expiry and alerting. PagerDuty maintenance windows and AWS lifecycle hooks are both time-boxed by design, with an explicit default action if nothing resolves the wait state in time. Consul's maintenance mode and Patroni's pause are not, and both ecosystems carry operational folklore about forgetting to clear them. A transition orchestrator should prefer the bounded model: every open transition record should have a maximum lifetime, past which it either auto-resolves to a safe default or raises an alert — never silently persists forever.
Testing the fence, not just the happy path. The patterns above are each individually simple; what causes real incidents is the interaction between them under restart, crash, or timing pressure. Useful test scenarios drawn directly from the failure modes above: inject a probe or health check that was already scheduled and pending before a transition record opens, and verify its result is discarded rather than acted on when it resolves after the fact; kill and restart the transition orchestrator itself mid-transition and verify the old health engine does not take over lifecycle ownership in the gap, and that the restarted orchestrator correctly resumes from the open transition record rather than treating it as abandoned; and simulate the transition record's expiry path to confirm it alerts rather than silently reverting to "nothing is wrong."
Recommendations for Agent Session Supervisors
A persistent agent runtime — one that keeps long-lived LLM agent sessions alive in a terminal multiplexer, supervised by a health engine that sends heartbeats and kills/restarts on missed or stale ones, with an inbound-message path that short-circuits to "delivered, retry later" when a session is marked unhealthy — sits squarely inside the pattern surveyed above, and inherits the same bug class Kubernetes has had to guard at several call sites across several releases.
-
Make the transition record the single source of lifecycle truth, with exclusive authority while open. Modeled on Patroni's pause and Orchestrator's
begin-downtime: once a migration, mode switch, or maintenance operation opens a transition record for a session, the steady-state health engine must treat that session as entirely out of its jurisdiction — not merely deprioritized — until the record closes. -
Tag every probe and heartbeat with the current transition generation; discard stale-generation results outright. Rather than trying to retroactively cancel a heartbeat probe dispatched moments before a transition opened, give every transition an epoch number and have the health engine compare an incoming heartbeat's epoch to the current one before acting — a result for a generation that no longer exists is discarded unevaluated.
-
Check "is a transition open?" at the moment of the kill/restart decision, not only when the probe was scheduled. This is the design reading we take from Kubernetes'
DeletionTimestamphistory, not something its bug reports prove: disabling new probes during a transition is necessary but not sufficient, since a probe already in flight can still resolve after the transition opens and, if unchecked, could still trigger remediation. The check belongs immediately before the kill/restart action executes, every time. Scope it to that destructive action, as Kubernetes does by stopping liveness and startup probes while letting readiness keep reporting: heartbeats can still be observed and recorded as status during a transition; they just may not drive a kill or restart. -
Give "not running by design" its own first-class state, distinct from "unhealthy." A session intentionally stopped mid-switch is not the same condition as one that crashed, and the state machine should make that a structural distinction — as OTP's
shutdown/normalexit reasons and systemd's stop-job exemption do — not an inferred special case layered on a single "unhealthy" flag. -
Queue inbound messages during an open transition instead of short-circuiting to "delivered, retry later." Marking a message delivered mid-transition is strictly worse than deferring it: it discards a real message under the fiction that it was handled. The PagerDuty/AWS lifecycle-hook pattern — hold and retry, don't discard — applies directly: the inbound receiver should check the same transition record and queue messages for replay once the transition closes.
-
Bound every transition with an expiry and an alert, never a silent indefinite state. Prefer the PagerDuty/AWS lifecycle-hook model over the Consul/Patroni one: give every open transition record a maximum duration, and if it has not closed within that window, auto-escalate to an operator alert rather than leaving sessions in limbo with no one watching the clock.
-
Test the fence explicitly, not just the transition's happy path. Inject a pending heartbeat probe immediately before opening a transition and confirm its late result is discarded; kill and restart the transition orchestrator mid-transition and confirm the old health engine does not reassert control in the gap, and that the restarted orchestrator resumes the same open record rather than abandoning it; force a transition past its expiry window and confirm it alerts rather than silently reverting.
The throughline across every system in this survey, from a fifteen-year-old init system to a 2022 Kubernetes issue report, is the same: remediation logic that only asks "have I heard from it lately?" will always eventually fight a plan it was never told about. The fix is never to make the health check smarter about guessing intent — it is to make intent an explicit, fenced, checked-at-action-time fact that the health check is required to consult before it is allowed to act.
Sources
- Kubernetes — Liveness, Readiness, and Startup Probes
- Kubernetes — PR #98571: stop probing a pod during graceful shutdown
- Kubernetes — PR #105215: terminate probes on node graceful shutdown
- Kubernetes — Issue #107473: liveness probe failures during pod termination and PR #107514 (proposed fix, closed unmerged)
- Kubernetes — PR #110191: re-enable readiness probes on termination; pod probes handled by pod worker
- Kubernetes — Issue #122824: readiness probes run during termination and the maintainer's closing comment (expected behavior)
- Kubernetes — Issue #42360: community discussion of probe behavior during termination
- Kubernetes — Disruptions (Pod disruption budgets)
- Kubernetes — Specifying a Disruption Budget for your Application
- Kubernetes — Safely Drain a Node
- systemd — systemd.service(5) man page
- systemd — sd_notify(3) man page
- Erlang/OTP — Supervisor Behaviour design principles
- Erlang/OTP — supervisor module reference
- HashiCorp Nomad — node drain command
- HashiCorp Nomad — reschedule block
- HashiCorp — Advanced Node Draining in HashiCorp Nomad
- HashiCorp Consul — consul maint command reference
- AWS — Suspend and resume Amazon EC2 Auto Scaling processes
- AWS — Considerations for suspending processes
- AWS — Temporarily removing instances from your Auto Scaling group (standby)
- AWS — EC2 Auto Scaling instance scale-in protection
- AWS — Amazon EC2 Auto Scaling lifecycle hooks
- Prometheus — Alertmanager documentation
- PagerDuty — Event Management documentation
- PagerDuty — Auto-Pause Incident Notifications
- Patroni — patronictl documentation (pause/resume)
- Orchestrator (openark) — topology-recovery.md
- Orchestrator (openark) — downtime.go source
- Orchestrator (openark) — Issue #1417: downtime vs. maintenance mode
- GitHub — October 21 Post-Incident Analysis

