Restart Backoff That Measures Time, Not Ticks
Executive Summary
Every mature process supervisor — Kubernetes' kubelet, systemd, Erlang/OTP, PM2, supervisord, Docker — specifies restart backoff in units of time: wait 10 seconds, then 20, then cap at 5 minutes; reset the count after the process has run cleanly for 10 minutes. None of them specify it in units of "checks." That distinction sounds pedantic until a supervisor's control loop stops running at a fixed period. A monitoring loop that polls once a second can get away with treating "60 iterations" as a stand-in for "60 seconds," because the two numbers are almost always equal. The moment that loop's period becomes variable — because it adapts to load, because a GC pause stretches an iteration, because the process was suspended and resumed — the stand-in silently stops matching the thing it stands in for, and every duration the supervisor thinks it's enforcing (backoff delay, stability window, crash-loop detection threshold) drifts by whatever factor the loop period drifted by.
This article traces how five production-grade supervisors encode backoff and reset thresholds, shows the specific mechanisms (timer drift, adaptive polling, suspend/resume, slow iterations) that break a tick-counted implementation, and sets out the engineering practice — monotonic clocks, injectable time sources, and a negative-control test that fails on tick-counting code — that keeps a time-based contract actually time-based as the surrounding scheduler evolves. The motivating case is a supervisor for long-lived AI agent sessions: restart backoff computed as min(5·2^n, 60) seconds with a reset after 60 seconds of stable running, originally checked by a fixed ~1-second loop and counted in "checks not running," then redesigned so that session-starting moves into a dispatcher whose scheduler polls adaptively between 1 and 3 seconds per round. Under the old counting method, that redesign would have silently stretched every backoff and reset window by up to 3× — a regression with no error message, no crash, and no symptom until someone timed a crash loop with a stopwatch and found it three times slower than configured.
The mechanism: why "one second, roughly" stops being roughly one second
A supervisor loop of the simplest kind looks like this:
while (true) {
checkAndMaybeRestart();
await sleep(1000);
}
Two different things are true about this loop, and the bug class lives in the gap between them. The intended period is 1000 ms. The actual period is 1000 ms plus however long checkAndMaybeRestart() took to run, because sleep is scheduled after the work completes, not on a fixed wall-clock cadence. If checkAndMaybeRestart() normally takes a few milliseconds, the gap is negligible and a reasonable engineer rounds it to zero. But "normally" is doing a lot of work in that sentence. Any of the following turns the gap from negligible to the dominant term:
- The loop itself becomes adaptive. A scheduler redesigned to poll less aggressively when there's nothing to do — a very ordinary, well-motivated change to reduce idle CPU and wake-ups — might widen its own sleep from 1 s to 3 s under light load. Code that was counting iterations now counts something that represents between 1 and 3 seconds of real time, interchangeably.
- A single iteration takes a long time. A slow filesystem check, a blocked
tmuxquery, a synchronous JSON parse of a large log, or a garbage-collection pause in the host runtime can make one "tick" take tens or hundreds of milliseconds longer than its neighbors — and if iterations occasionally take seconds, the count-based model of "60 ticks = 60 seconds" degrades further. - The process hosting the loop is suspended. A laptop closing its lid, a VM pausing for a hypervisor-level snapshot, or a container being checkpointed all stop the loop entirely for an interval the loop never observes as elapsed time unless it's explicitly told to look.
- The loop is intentionally throttled under backpressure, e.g., reduced polling frequency when rate-limited by an upstream API — exactly the kind of condition an AI agent supervisor needs to survive gracefully rather than have its own internal time sense distorted by.
None of these failure modes announce themselves. A tick-counted backoff that's supposed to wait 60 seconds and actually waits 180 seconds doesn't throw, doesn't log an error, and doesn't fail any test that doesn't specifically simulate the loop period changing. It just restarts a crashed process three times slower than the operator configured, or — the more dangerous direction — resets its stability counter three times faster than intended if the miscount runs the other way, letting a process that's crash-looping every 40 seconds get miscounted as "stable" and have its backoff prematurely reset to the minimum, causing the exponential curve to restart from the bottom indefinitely. On an agent supervisor, that specific failure mode is actively dangerous: a config or auth error that always crashes the agent shortly after each restart should drive the backoff up toward its ceiling, not get reset to five seconds forever because a reset threshold got satisfied on a miscounted clock.
How mature supervisors define it: always in time, never in ticks
The specifications below are drawn from each project's own source or documentation. The pattern is uniform: thresholds are durations; the mechanism used to measure those durations is the designer's own problem to solve correctly, and notably, every one of them that publishes implementation detail uses either real elapsed-time comparisons or an explicit, swappable clock abstraction — never a raw iteration count.
| Supervisor | Backoff curve | Reset / stability condition | Startup-failure limit |
|---|---|---|---|
| Kubernetes kubelet (CrashLoopBackOff, pre-KEP-4603 default) | 10s, 20s, 40s, 80s, 160s, capped at 300s (doubling) | Counter resets after the container has run for 10 minutes (2× the max backoff) without exiting | No separate limit; backoff alone governs restart rate |
Kubernetes kubelet (ReduceDefaultCrashLoopBackoffDecay feature gate, KEP-4603) | 1s initial, capped at 60s (doubling) | Reset threshold held at a flat 10 minutes, decoupled from the max-backoff value | Same |
| systemd (≥ v254) | RestartSec as the base, growing over RestartSteps steps up to RestartMaxDelaySec | No "ran successfully" reset of the restart delay itself; instead StartLimitIntervalSec sets a time interval and StartLimitBurst the max starts allowed within it (the implementation opens the interval at a start and resets the whole count once it expires: a fixed window, not a rolling history of starts) | Unit enters failed (rate-limit tripped) once StartLimitBurst is exceeded inside StartLimitIntervalSec |
| Erlang/OTP supervisor | No backoff curve by default — a child is restarted immediately per its restart type (permanent/transient/temporary) | intensity restarts allowed per period seconds (default: 1 restart per 5 seconds); exceeding it shuts the whole supervisor down | Supervisor (and its children) terminate if intensity is exceeded within period |
| PM2 | exp_backoff_restart_delay: base delay × 1.5 per attempt (e.g. 100ms → 150ms → 225ms → …), up to a 15000ms maximum | Delay resets to its base value after more than 30 seconds of uptime without restarts | max_restarts consecutive restarts faster than min_uptime (default 1000ms) before giving up (default max: 16) |
| supervisord | No built-in exponential backoff; fixed restart on exit per autorestart policy | startsecs: process must stay in RUNNING for N seconds to count as a successful start (default 1s) | startretries consecutive failed starts before FATAL (default 3) |
Docker (--restart on-failure) | 100ms base, doubling (100ms, 200ms, 400ms, 800ms, …), capped at 1 minute | Backoff delay drops back to 100ms after any run of at least 10 seconds; the restart count that on-failure:N checks is separate, is not reset by a long run, and is zeroed when the container is started again manually (docker start / docker restart) | Optional on-failure:N caps total restart count |
| runit / s6 | No exponential curve; s6 and runit both enforce a flat ~1-second minimum delay between spawns if a service exits in under 1 second | N/A — philosophy is "a crashing service is an operator problem, not something to paper over with backoff tuning" | None built in; left to external supervision |
A few things are worth pulling out of this table individually.
Kubernetes' own justification for its reset threshold is the single most on-point primary source for this whole topic. The KEP-4603 proposal to reduce kubelet's default max backoff from 5 minutes to 1 minute deliberately kept the 10-minute reset threshold untouched, and the stated reason is exactly the tick-granularity problem this article is about, stated from the other side: "the granularity of the associated worker polling loops governing restart behavior is between 1 and 10 seconds, so a reset value under 10 seconds is effectively meaningless." In other words, Kubernetes' own authors reasoned explicitly about the relationship between a stability threshold (how long must a thing run before we trust it) and the polling granularity of the loop that observes it — and concluded you cannot set a meaningful threshold below the loop's own measurement resolution. That's the generalizable principle: a time-based contract is only as precise as the clock enforcing it, and if that "clock" is actually a loop counter, its resolution is not 1 second, it's "however long the slowest iteration during the observation window happened to take."
systemd's split between RestartSec/RestartSteps and StartLimitIntervalSec/StartLimitBurst separates two different questions that are easy to conflate: how long to wait before the next restart (a growing delay), versus how many starts are tolerated within a time window before giving up entirely (a rate limit). Both are time-denominated, but they answer different questions, and an agent supervisor copying this pattern should keep them as two separate knobs rather than trying to derive a hard stop from the backoff curve alone — a backoff curve capped at 60 seconds, left running forever, will retry forever at a tolerable rate; only a separate attempt-count-within-a-window rule turns "slow but infinite" into "give up and alert a human."
Erlang/OTP's intensity/period pair is the oldest version of this idea in the list (the design dates to the 1990s telecom systems OTP was built for) and is explicit that both numbers are a rate — restarts per unit time — not a count in isolation. A supervisor configured for intensity: 1, period: 5 is not saying "allow one restart"; it's saying "allow a restart rate of up to 1 per 5 seconds," which only makes sense as a time-windowed measurement.
PM2's two mechanisms — exp_backoff_restart_delay and min_uptime/max_restarts — are a particularly close analogue to an AI agent supervisor, because PM2 is solving the identical problem (user-space process supervision of a long-lived script that might fail transiently or might be stuck in a bad state). min_uptime is the closest PM2 setting to what this article calls the stability window: the declaration page describes it as the "min uptime of the app to be considered started (defaults to 1000 ms)", and max_restarts as the "number of consecutive unstable restarts (less than 1sec interval or custom time via min_uptime)". A bare number is read as milliseconds, and strings such as "10s" or "5m" set other units. That is a duration contract; the page does not say which clock PM2's implementation reads to enforce it.
Tick-counting vs. elapsed-time: the actual failure modes
The category of bug is familiar wherever a repeated-poll loop is treated as a clock, but it has specific, well-documented sub-cases worth separating:
1. setInterval and a self-scheduling loop misbehave in different ways when the work is slow. A setInterval(fn, 1000) timer does not queue one invocation for every period that elapsed while fn was running. Node re-inserts the repeating timer once, after each callback returns, with its next deadline computed from a timestamp taken just before that callback ran; the HTML standard likewise re-runs the timer initialization steps once after the handler runs. So when a synchronous fn blocks past the interval, the missed periods are skipped rather than replayed: the next call comes as soon as the event loop is free, and the rhythm resumes from there. (On Node v24, a 20 ms interval whose callback blocked for 130 ms produced one gap of about 130 ms followed by gaps of about 20 ms, not a run of catch-up calls; exact timing is runtime-specific.) Asynchronous work is a separate case: if fn starts an async operation and returns, setInterval keeps firing on schedule while earlier operations are still in flight, so slow async work overlaps with itself instead of delaying the timer. A self-scheduling pattern (setTimeout that re-arms itself inside its own callback, i.e. fn(); setTimeout(fn, 1000)) does not overlap as long as fn does its work synchronously. If fn starts asynchronous work and returns, that work still overlaps once it runs longer than the delay, just as under setInterval; the asynchronous form that avoids overlap waits for the work before re-arming (await fn(); setTimeout(loop, 1000)). In either non-overlapping form the loop drifts later every time the work takes non-zero time, because the next tick's countdown starts only once the current work finishes — turning a nominal 1-second loop into something closer to "1 second plus however long the work took," compounding every iteration. Most real supervisor loops (including a setTimeout-after-each-iteration pattern) use the second model, so their drift is cumulative lateness — which is the dangerous direction for backoff math, because it means the loop silently gets slower than its nominal period and every tick-counted duration silently gets longer than configured, with no compensating correction anywhere in the design.
2. Adaptive polling is the drift mechanism turned into an intentional feature, and it is the one most directly relevant here: a scheduler that polls every 1 second when busy and backs off to every 3 seconds when idle is a sound, common design (less wasted CPU and wake-ups, same responsiveness when it matters) — but it means the very same counted value ("60 polling rounds") can represent anywhere from 60 to 180 seconds of real time depending on load at the time, a 3× range with no bound on which end a given run lands on.
3. Wall clock vs. monotonic clock is a second, separate trap that elapsed-time fixes can walk straight into if they pick the wrong clock. Date.now() (or any other wall-clock read) can jump — NTP corrections, manual clock changes, leap-second smoothing — and can even run backward, which breaks any duration computed as end - start if the system clock was adjusted between the two reads. The fix used throughout the industry is a monotonic time source: Node's process.hrtime() / performance.now(), or any OS-level CLOCK_MONOTONIC read — a clock that is not affected by discontinuous jumps in the system time (on Linux it is still slewed by NTP's incremental adjustments) and whose consecutive reads never go backward. Go builds this into its ordinary idiom: the Time returned by time.Now() carries both a wall clock reading and a monotonic clock reading, and comparisons and subtractions between two such values — t.Sub(start), time.Since(start), time.Now().Before(deadline) — use the monotonic readings. The Go trap is losing that reading before subtracting: t.Round(0), t.UTC(), t.Unix(), and any Time that was serialized and parsed back keep only the wall reading, and a subtraction involving such a value falls back to the wall clock. The rule of thumb used across these ecosystems: wall-clock time is for timestamps meant for humans or logs ("this happened at..."), monotonic time is for durations and timeouts ("how long since...").
4. Monotonic clocks have their own edge case: suspend/resume. On Linux, CLOCK_MONOTONIC is explicitly defined to not count time the system is suspended — if a laptop sleeps for ten minutes, a duration measured on it across that sleep comes out ten minutes shorter than a suspend-inclusive measurement. CLOCK_BOOTTIME (since Linux 2.6.39) is identical except that it also includes suspended time, and Go's documentation warns that on some systems its monotonic clock likewise stops while the computer sleeps. For a long-lived desktop or laptop-hosted agent supervisor — plausible for local developer tooling, less likely for a cloud-hosted one — this forces a policy decision before a clock choice: should time spent suspended count toward a backoff delay or a stability window? If the contract is suspend-inclusive ("wait 60 seconds of real time"), a CLOCK_MONOTONIC deadline errs late, never early: a backoff with four seconds left when the machine sleeps still has four monotonic seconds to wait after wake, even if far more than its full delay has passed in real time, and a stability window spanning the sleep is under-credited, so the reset arrives later than the contract says. If the contract counts only time the machine was awake to observe the process, CLOCK_MONOTONIC is the right clock, and it is CLOCK_BOOTTIME that would let a backoff delay or stability window expire entirely during sleep. The practical choice is to decide which of these two policies the supervisor implements and pick the clock that matches it, rather than defaulting to whichever one the standard library makes most convenient.
Defining "stable": from-start vs. from-ready, and why it matters for cold-start-heavy agents
A second, related ambiguity sits inside "reset the backoff after the process has run stably for N seconds": stable from when? Two candidate start points give materially different behavior:
- From process start. The clock starts the instant the supervisor spawns the process. This is simple and is what most of the table above actually measures (kubelet's 10-minute reset, PM2's
min_uptime), but it conflates "took a while to initialize" with "crashed quickly" if initialization itself is slow and variable — a concern Kubernetes addressed separately with the startup probe, which exists precisely to stop the liveness probe from punishing a container for a slow-but-healthy initialization sequence by disabling liveness/readiness checks entirely until the startup probe first succeeds. - From readiness. The clock starts only once the process has signaled it's actually doing useful work — a readiness probe passing, a login/auth handshake completing, a model finishing its cold-start load. This is the more correct measure of "is this instance actually stable," but it requires the supervised process to expose an explicit readiness signal, which not every process does.
For an AI agent supervisor specifically, this distinction maps onto a known, named failure split: a session that never became ready (crashed during an expensive cold start — model/context load, auth handshake, tool-server attach — before doing any real work) versus one that crashed after ready (ran successfully for a while, then failed mid-task, possibly due to a transient upstream rate limit or a bug triggered by specific input). These call for different operator responses — "never ready" points at config, credentials, or environment and should probably alert a human well before the backoff ceiling is reached, since retrying a broken config at increasing delays just delays the inevitable diagnosis — while "crashed after ready" is closer to the transient-failure case backoff is actually designed for. A supervisor that only measures "ticks since start" cannot distinguish these two cases at all; one that measures elapsed time and tracks whether a readiness signal was ever observed can.
Jitter: when it matters, and when it's a distraction
The AWS Architecture Blog's "Exponential Backoff And Jitter" is the canonical reference here, and it's worth being precise about what problem jitter solves, because it's easy to import the pattern somewhere it doesn't help. The post works through three variants:
- Full Jitter:
sleep = random(0, min(cap, base * 2^attempt))— the entire delay is randomized up to the exponential ceiling. - Equal Jitter: half the exponential delay is kept fixed, the other half randomized — a compromise that avoids ever sleeping near zero.
- Decorrelated Jitter: each retry's delay is randomized based on the previous delay rather than purely on attempt count.
The post's simulation doesn't pick one winner on every measure. Equal Jitter comes out worst of the three: "It does slightly more work than 'Full Jitter', and takes much longer." Decorrelated makes the most calls: "the number of calls is approximately the same for 'Full' and 'Equal' jitter, and higher for 'Decorrelated'." Between the remaining two, the post says it's a trade-off: "The decision between 'Decorrelated Jitter' and 'Full Jitter' is less clear. The 'Full Jitter' approach uses less work, but slightly more time." So Full Jitter costs the contended resource fewer calls, and Decorrelated finishes slightly sooner. Per the post, both cut client work and server load substantially compared with un-jittered exponential backoff.
The mechanism jitter fixes is synchronized retries across many independent clients hitting the same shared resource — thousands of SDK clients all backing off from a throttled API and then all retrying at the exact same moment because they all compute the exact same exponential delay, turning backoff into a self-reinforcing thundering herd. That mechanism simply doesn't exist for a supervisor managing a single long-lived agent session restarting against its own dependencies: there's no fleet, no shared contended resource, no other client computing the same delay at the same instant to collide with. Jitter is a reasonable, cheap thing to add regardless, but the specific justification is different: it exists to avoid a resonance between the backoff curve and some unrelated periodic process (a cron job, another timer in the same host) that happens to line up with it, not to protect a shared backend from a stampede. Where it does become essential for an agent supervisor is the moment there are many independent agent sessions sharing a rate-limited upstream API (the classic AWS motivating case reappears exactly), at which point the fleet-level reasoning fully applies again.
Testing a timing contract: the negative control
A backoff/reset implementation is a timing contract: "wait at least X, at most Y; reset after Z of stability." The natural way to test it is also the way most teams get it wrong — running the real implementation against real timers and asserting on wall-clock behavior, which makes the test either slow (if it actually waits) or non-deterministic (if it doesn't wait quite long enough and flakes under CI load).
The fix used throughout the ecosystems surveyed here is to make time itself an injectable dependency rather than a global ambient call:
- JavaScript/Node: Sinon's fake timers (
@sinonjs/fake-timers, used viasinon.useFakeTimers()) replace the globalsetTimeout,setInterval,Date, and evenperformance.now()/process.hrtimewith a controllable virtual clock. Tests advance time synchronously withclock.tick(ms)and assert on what fired, with no real waiting and no flakiness tied to CI machine speed. - Go: the
benbjohnson/clockpattern — now continued under community maintenance — defines aClockinterface with aNew()(real) andNewMock()(test) implementation; application code takes aClockas a dependency instead of callingtime.Now()/time.Sleep()directly, and tests drive the mock clock forward explicitly withAdd(duration). - Kubernetes itself does exactly this for its own backoff library.
client-go/util/flowcontrol'sBackoffstruct holds aclock.Clockfield, and the package exposesNewFakeBackOffspecifically to inject a test clock — meaning the same codebase that documents the backoff algorithm in prose also treats the algorithm's dependence on a clock as a first-class, swappable interface in code, not an implicit global.
The test design that actually catches a tick-for-time substitution bug is a negative control: before trusting that a test proves the fix works, first prove the test can fail. Run one assertion, unchanged, against a deliberately wrong tick-counting implementation and confirm it fails. Then run that same assertion against the time-based implementation and confirm it passes. For that to mean anything, the assertion has to encode the timing contract itself, not the output of either implementation. For the motivating redesign in this article, the contract is:
- Delay. After the k-th consecutive failed restart attempt (k = 0 for the first crash), the supervisor waits
min(5·2^k, 60)seconds before the next restart, measured from when it observed the process down. - Reset. A process that stays up for 60 seconds resets k to 0.
- Tolerance. A restart may come at most one polling period late, because the supervisor only acts when it polls.
Two inputs need to stay separate here. The attempt count k selects which delay applies. Elapsed time decides when that delay has expired and when the stability window has been met. A test that only fixes elapsed time doesn't pin down the attempt count. If a restarted process crashes as soon as it starts, a supervisor that polls every second makes more restart attempts in a given minute than one that polls every three seconds, so the two legitimately reach different backoff values. The test below therefore fixes the attempt history: it scripts how long each restarted process stays up, so every polling cadence sees the same attempts with the same outcomes. Only the polling rhythm changes between runs.
const test = require('node:test');
const assert = require('node:assert');
// The contract: after the k-th consecutive failed restart attempt (k = 0 for
// the first crash), wait min(5 * 2^k, 60) s, measured from when the supervisor
// observed the process down, before the next restart. The attempt count k
// selects the delay; elapsed time decides when that delay has expired. A
// process that stays up for 60 s resets k to 0.
const BASE_S = 5, CAP_S = 60, STABLE_S = 60;
const delayS = (k) => Math.min(BASE_S * 2 ** k, CAP_S);
// Broken: one poll is treated as one second. Correct only if every poll
// really does take one second.
function tickCountingSupervisor() {
let k = 0, downPolls = 0, upPolls = 0;
return {
poll(running) { // returns true if it restarts now
if (running) {
downPolls = 0;
if (++upPolls >= STABLE_S) k = 0;
return false;
}
upPolls = 0;
if (downPolls < delayS(k)) { downPolls++; return false; }
k++; downPolls = 0;
return true;
},
};
}
// Repaired: reads an injected monotonic clock (performance.now() in production).
function elapsedTimeSupervisor(now) {
let k = 0, downSince = null, upSince = null;
return {
poll(running) {
const t = now();
if (running) {
downSince = null;
if (t - upSince >= STABLE_S * 1000) k = 0;
return false;
}
if (downSince === null) downSince = t;
if (t - downSince < delayS(k) * 1000) return false;
k++; downSince = null; upSince = t;
return true;
},
};
}
// Fake clock + scheduler. The outcome of each restart attempt is scripted
// (seconds the restarted process stays up), so every cadence sees the same
// attempts and the same outcomes; only the polling rhythm differs.
function simulate(makeSupervisor, cadenceMs, upForS) {
let t = 0, upUntil = 0, downSeen = null, attempt = 0, round = 0;
const sup = makeSupervisor(() => t);
const gapsS = []; // observed-down -> restart, per attempt
while (attempt < upForS.length) {
const running = t < upUntil; // process is down at t = 0
if (!running && downSeen === null) downSeen = t;
if (sup.poll(running)) {
gapsS.push((t - downSeen) / 1000);
upUntil = t + upForS[attempt++] * 1000;
downSeen = null;
}
t += cadenceMs[round++ % cadenceMs.length];
}
return gapsS;
}
const UP_FOR_S = [0, 90, Infinity]; // attempt 1 dies at once, 2 runs 90 s, 3 stays up
const EXPECTED_DELAY_S = [5, 10, 5]; // k = 0, then k = 1, then reset by the 90 s run
const CADENCES_MS = [[1000], [3000], [1000, 3000, 3000]]; // fixed 1 s, fixed 3 s, adaptive
function checkTimingContract(makeSupervisor) {
for (const cadenceMs of CADENCES_MS) {
const gapsS = simulate(makeSupervisor, cadenceMs, UP_FOR_S);
const slackS = Math.max(...cadenceMs) / 1000; // can't act before the next poll
assert.ok(
EXPECTED_DELAY_S.every((d, i) => gapsS[i] >= d && gapsS[i] <= d + slackS),
`cadence ${cadenceMs} ms: waited [${gapsS}] s, contract [${EXPECTED_DELAY_S}] s (+${slackS} s)`);
}
}
test('negative control: the tick-counting supervisor breaks the contract', () => {
assert.throws(() => checkTimingContract(tickCountingSupervisor), assert.AssertionError);
});
test('the elapsed-time supervisor meets the same contract', () => {
checkTimingContract(elapsedTimeSupervisor);
});
Both tests run the same assert.ok in checkTimingContract. Only the implementation passed in changes. Run with node --test, the measured gaps between "observed down" and "restarted" are:
| Cadence | Tick-counting | Elapsed-time | Contract (+ slack) |
|---|---|---|---|
| fixed 1 s | 5, 10, 5 s | 5, 10, 5 s | 5, 10, 5 s (+1 s) |
| fixed 3 s | 15, 30, 60 s | 6, 12, 6 s | 5, 10, 5 s (+3 s) |
| adaptive 1/3/3 s | 11, 22, 46 s | 7, 10, 7 s | 5, 10, 5 s (+3 s) |
The fixed-1-second row is why this bug survives in production: at that cadence the broken implementation produces exactly the right numbers, and any test that only runs the old loop passes. At a 3-second cadence the assertion fails on the tick-counting version with cadence 3000 ms: waited [15,30,60] s, contract [5,10,5] s (+3 s). Both bugs show up at once. The first two delays are stretched 3× (15 s and 30 s instead of 5 s and 10 s). And the 90-second run is seen as up on only 29 polls, which never reaches the "60 polls" stability threshold, so k is not reset and the third delay climbs to 60 s (20 polls × 3 s) instead of 5 s. The elapsed-time version stays inside the window at every cadence.
The invariant this test guards should be stated with its assumption attached. For a fixed attempt-and-outcome history, each restart happens no earlier than the contracted delay and no later than that delay plus one polling period, whatever the polling cadence. It does not say that equal elapsed time produces equal backoff. Across different cadences, the number of restart attempts made in a given stretch of time can legitimately differ, and so can the backoff level reached. The one-period tolerance is also the honest form of the assertion for a real adaptive scheduler: the supervisor can't detect a state change faster than its own next poll. That is the KEP-4603 reasoning about reset thresholds applied as a test tolerance instead of a design constant.
Recommendations for an AI agent supervisor
Generalizing from the comparison table and the failure mechanisms above, for a supervisor restarting long-lived agent sessions (expensive cold starts, rate-limited upstream APIs, failures that range from transient network blips to permanently broken config):
- Specify backoff and reset purely in elapsed time, and compute that elapsed time from a monotonic clock, never from a count of polling rounds and never from wall-clock subtraction. If the deployment target includes machines that suspend (laptops, some VM hosting), decide whether suspended time should count: if it should, use a suspend-inclusive clock (
CLOCK_BOOTTIME-equivalent); if it should not,CLOCK_MONOTONICalready leaves it out. - Decouple the polling/scheduling loop's period from every duration the supervisor enforces. A scheduler is free to poll adaptively (1–3 s, or wider) for efficiency; backoff delay, stability-window length, and crash-loop detection must all be read from a clock at decision time, not inferred from how many times the loop happened to run.
- Track whether a readiness signal was ever observed, separately from "process is running." This is what lets the supervisor distinguish never became ready (cold start or config/auth failure — a case worth surfacing to a human quickly rather than retrying indefinitely at increasing delay) from crashed after ready (plausibly transient — the case exponential backoff is designed for).
- Keep the backoff-delay curve and the hard-stop/alert threshold as two separate mechanisms, following systemd's
RestartSecvs.StartLimitBurst/StartLimitIntervalSecsplit: a capped exponential delay alone degrades gracefully into "retry forever, slowly," which is fine for a human-supervised system but insufficient alone for an unattended one — pair it with a restart-count ceiling (OTP'sintensityrestarts perperiod, or PM2'smax_restartsconsecutive runs shorter thanmin_uptime) that stops retrying and escalates once a session is clearly looping rather than recovering. - Set stability-window and reset thresholds no finer than the scheduler's own worst-case polling period, and document why, following KEP-4603's explicit reasoning: a reset value finer than the loop's measurement resolution is not actually controllable and just adds false precision to the configuration.
- Treat jitter as optional for a single supervised session and necessary the moment multiple sessions share a rate-limited upstream. Don't add jitter reasoning (fleet-level thundering-herd avoidance) to justify a design choice that's actually solving a different, single-process problem — but revisit it the moment the architecture adds concurrent sessions against the same API.
- Test the timing contract with an injectable/fake clock, and include a negative control. Before trusting a test suite's green result on the time-based implementation, confirm the same assertions, unchanged, go red against a deliberately tick-counting implementation when the polling period differs from the one the tick math assumes (a fixed 3 s or adaptive 1–3 s schedule, not just the old fixed 1 s loop), with the attempt history held fixed — only then does a passing suite mean what it's supposed to mean.
Sources
- Kubernetes documentation: Pod Lifecycle — restart backoff
- KEP-4603: Tune CrashLoopBackoff
- kubernetes/enhancements issue #4603: Tune CrashLoopBackOff
- kubernetes/enhancements issue #5593: Configure the max CrashLoopBackOff delay
- Tunable CrashLoopBackOff in GKE (Google Cloud / DEV Community)
- client-go util/flowcontrol backoff.go (source)
- client-go util/flowcontrol backoff_test.go (source, fake clock usage)
- Kubernetes: Configure Liveness, Readiness and Startup Probes
- systemd.service(5) manual page (Debian)
- systemd.unit(5): StartLimitIntervalSec= / StartLimitBurst=
- systemd v257 ratelimit.c (start-limit window implementation)
- "How systemd exponential restart delay works"
- Erlang System Documentation: Supervisor Behaviour
- Erlang/OTP
supervisormodule reference - PM2 — Ecosystem File / application declaration (exp_backoff_restart_delay, min_uptime, max_restarts)
- PM2 — Restart Strategies (exponential backoff restart delay)
- Supervisor (supervisord) Configuration File documentation
- s6-supervise manual (skarnet.org)
- "Back off" setting for crashing services with s6+openrc? (supervision mailing list)
- Docker restart policies — "From PM2 to Docker: Automatic Restarts"
- Moby v28.5.1 restartmanager.go (restart delay and reset)
- AWS Architecture Blog: "Exponential Backoff And Jitter"
- AWS Builders' Library: "Timeouts, retries, and backoff with jitter"
- "Why Javascript timer is unreliable, and how can you fix it" (setInterval vs setTimeout drift)
- Node.js v24.11.0 lib/internal/timers.js (repeating timer re-insertion)
- HTML Standard: Timers
- MDN: Performance.now()
- Go
timepackage: Monotonic Clocks - clock_gettime(3) — Linux manual page (CLOCK_MONOTONIC, CLOCK_BOOTTIME)
- "Your Clock Can Go Backward—Use the Right One for Durations" (monotonic vs wall clock)
- CLOCK_MONOTONIC vs CLOCK_BOOTTIME and suspend behavior (eclipse-paho/paho.mqtt.c issue #1095)
- Baeldung: Understanding Timekeeping and Clocks in Linux
- Sinon.JS: sinon.useFakeTimers()
- sinonjs/fake-timers (GitHub)
- benbjohnson/clock — mocking time in Go (GitHub)
- Wikipedia: Negative testing

