The Energy Cost of Agentic Inference: Why Continuously Running Agent Fleets Break the Old Power Math
Executive Summary
Every energy-consumption analysis of AI written through early 2026 shared an implicit assumption: a "query" is one forward pass, or at most one short exchange, and the energy cost is the cost of generating the reply tokens. That assumption quietly stopped being true the moment agents started making tool calls, and a KAIST paper published at HPCA 2026 ("The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective") puts a number on how wrong it now is: per the research team's own reporting, an agentic query can consume up to 136.5x more energy than a conventional single-turn chatbot query, with response time inflated up to 153.7x, and — the counterintuitive part — GPUs sitting idle for as much as 54.5% of total execution time while the agent waits on external tools, retrieval, or code execution to return. A 70-billion-parameter agent averaged 348.41 watt-hours per query in their measurements (KAIST News Center; TechXplore coverage). This is not a story about bigger models needing more FLOPs; it is a story about expensive, power-hungry accelerators being held in a waiting state by a workload shape nobody designed the serving stack for.
This matters specifically for continuously running autonomous agent fleets — background research agents, monitoring loops, scheduled pipelines, always-on customer-facing assistants — because their cost profile inverts the assumptions that GPU serving infrastructure was tuned around for a decade. Human-driven chat is bursty: request arrives, generate, done, GPU goes idle until the next human types something, and infrastructure teams optimized for time-to-first-token and burst absorption. A fleet of autonomous agents that plan, call tools, wait, re-plan, and call more tools all day long produces a fundamentally different power curve: long-lived, low-utilization, memory-heavy sessions that keep a GPU "reserved" without keeping it busy. Related industry characterization work discussed alongside the KAIROS project (April 2026) frames the scale in similar terms, describing agentic inference as consuming roughly two to three orders of magnitude more power than single-turn LLM serving for comparable models — directionally consistent with, though not identical in methodology to, the KAIST measurements above.
2026 has produced the first wave of infrastructure specifically built to answer this: power-aware, context-aware serving systems (KAIROS reports 27% average, up to 39.8%, power reduction on real agentic workloads by treating agent context growth as a first-class scheduling signal), a shift in the industry's headline efficiency metric from tokens-per-second to tokens-per-watt, and renewed interest in GPU frequency capping and asynchronous, non-blocking agent execution patterns specifically to keep accelerators fed instead of idling on tool-call round trips. This article works through the measurement evidence, the architectural response, and what changes operationally when the workload is a fleet that never stops, not a chat session that ends.
Why Agentic Workloads Break the Old Energy Math
The KAIST HPCA 2026 measurements
The paper (DOI: 10.1109/hpca68181.2026.11408569, first author Jiin Kim, KAIST School of Electrical Engineering, led by Professor Minsoo Rhu) is, per KAIST's own announcement, the first systematic infrastructure-level characterization of agentic reasoning and test-time scaling as a distinct energy workload class, rather than treating "agent" as just "LLM plus a wrapper." Its headline figures, as reported by KAIST and corroborated in TechXplore's coverage:
| Metric | Conventional single-turn query | Agentic query | Multiplier |
|---|---|---|---|
| Energy per query | baseline | up to 136.5x baseline | ~136x |
| End-to-end latency | baseline | up to 153.7x baseline | ~154x |
| GPU idle time during execution | near-zero | up to 54.5% of wall-clock | — |
| Absolute energy (70B model, agentic) | — | 348.41 Wh/query average | — |
The team open-sourced their agent implementations and benchmark harness specifically so other labs could reproduce and extend the measurement methodology — an important detail, because prior energy-of-LLM-inference literature (e.g., "From Words to Watts," "Where Do the Joules Go?") measured single forward passes and simply did not have an agentic-loop benchmark to point at. This paper is the first to give the field a citable number for "how much more does dynamic, tool-interleaved reasoning cost, end to end, including the idle time."
Where the energy actually goes: waiting, not computing
The mechanism is not exotic. A ReAct-style agent loop is: think → call tool → wait for tool result → think → call tool → wait → ... → final answer. Every wait is time the GPU holding that agent's KV-cache state is not doing useful computation for that request, but — because the request's context must stay resident to resume generation — the accelerator can't simply be released to another job the way it could be if this were a stateless, one-shot completion. Multiply this by search steps, retrieval calls, subagent dispatches, and sandboxed code execution, and the "expensive GPU doing nothing" fraction climbs fast. The 2026 survey "Networking-Aware Energy Efficiency in Agentic AI Inference" frames this explicitly as a networking-and-orchestration problem as much as a model-efficiency problem: the tool round trip, not the transformer forward pass, is increasingly the dominant term in the energy budget of a complex agent turn.
A second, less obvious effect compounds this: naive power-saving via GPU frequency scaling backfires on agentic workloads specifically. KAIROS's authors found that lowering GPU clock frequency to save power during what looks like an idle window can push the system into a thrashing regime — because agent requests carry long-lived, growing context (the full conversation + tool history), reduced compute throughput at the same batch size causes memory pressure to spike, which degrades both latency and power efficiency simultaneously. In other words, the standard power-management trick that works fine for stateless single-turn serving (clock down when idle, clock up on demand) actively makes agentic serving worse if applied without context awareness. This is the core reason the field needed agent-specific serving systems rather than reusing single-turn power management.
Tokens-Per-Watt: The Metric Replacing Tokens-Per-Second
Through 2025, inference efficiency conversations centered on tokens-per-second-per-dollar and PUE (power usage effectiveness) at the facility level. Neither captures what 2026 operators actually need to optimize, because both ignore that power, not floor space or even GPU count, has become the binding constraint at most AI data center sites — reported build-outs are commonly described as running on the order of 100-750 MW per campus for hyperscale AI facilities, and grid interconnect queues, not chip supply, are widely characterized as the new bottleneck (Digitimes, Spheron, and Bloom Energy's 2026 Data Center Power Report are among the outlets converging on this framing).
The metric that has consolidated as the actionable target is tokens per watt: useful output tokens generated per unit of energy consumed, at the workload level rather than the chip-spec level. NVIDIA's own developer messaging in 2026 ("Scaling Token Factory Revenue and AI Efficiency by Maximizing Performance per Watt") frames the entire AI-factory economic model around this ratio — because in a power-constrained facility, revenue per megawatt is capped by tokens-per-watt, not by how many racks you can physically fit. The related "1/W Law" analysis (arXiv 2603.17280) formalizes how context length, routing topology, and GPU generation jointly determine the achievable tokens-per-watt ceiling for a given serving configuration — directly relevant to agents, whose context grows monotonically across a multi-step task in a way a single chat turn's does not.
GPU power capping: a real, quantified trade-off
One concrete lever operators are using: capping GPU power draw below its rated maximum. Results circulating in 2026 infrastructure literature suggest a rough shape for this trade-off: cutting a GPU's power cap by around 30% (illustrative figures put this near a 250W-to-175W step) appears to increase inference latency by roughly a mid-single-digit percentage, for a net energy reduction on the order of a fifth to a quarter per unit of work — treat the precise decimal figures as illustrative rather than a benchmark this article independently verified. For latency-insensitive workloads — which describes most background and continuously-running agents, as distinct from a human waiting on a chat reply — this is close to a free win: accept a single-digit percentage latency hit in exchange for a quarter less energy per completed task. The catch, per the KAIROS finding above, is that power capping must be applied context-aware, not blanket, or the thrashing effect eats the savings on agents with large, growing context windows.
Power-Aware Serving Systems Built for Agents
2026 produced the first cohort of serving-layer systems designed around agentic workload shape rather than adapted from single-turn LLM serving:
| System | Approach | Reported result |
|---|---|---|
| KAIROS (UIUC/Michigan, arXiv 2604.16682) | Tracks requests at agent granularity; uses live agent context (not just queue depth) as the control signal to jointly tune GPU frequency, per-instance concurrency, and multi-instance placement/routing | 27% average power reduction, up to 39.8%, while meeting latency SLOs, across software- and data-engineering agent benchmarks |
| GoodServe (arXiv 2605.16867) | Maximizes "goodput" (useful completed agent work per unit resource) for agentic inference over heterogeneous GPU/accelerator pools | Improves effective throughput per resource-dollar on mixed hardware fleets |
| Concur (arXiv 2601.22705) | Proactive, agent-level admission control — decides whether to admit a new agent task based on projected resource/context growth, rather than reactive queueing | Reduces resource contention and SLO violations under agentic batch load |
| SLO-aware disaggregated scheduling (arXiv 2605.02329) | Separates prefill and decode phases across pools to avoid request-imbalance-driven power waste under heterogeneous agent request shapes | Improves tail-latency and utilization stability |
The common thread across all four: none of them treat "an agent request" as equivalent to "a chat completion request." Each explicitly models the fact that an agent's resource footprint (context size, number of remaining tool calls, likely completion time) evolves over the life of the task, and that scheduling/power decisions made without that signal systematically under- or over-provision. KAIROS's own framing is the clearest statement of the paradigm shift: prior power-management techniques "focus almost entirely on single-turn LLM serving," and agentic serving "behaves fundamentally differently" because of long-lived, evolving context across tool-interleaved turns.
A representative (illustrative, not vendor-literal) shape of what an agent-context-aware power policy configuration looks like in this generation of systems:
# illustrative agent-aware serving policy (KAIROS-style control loop)
policy:
signal_source: agent_context # not just queue_depth / batch_size
controls:
gpu_frequency:
mode: context_adaptive # scale with tracked context growth rate, not idle/busy alone
floor_mhz: 1200 # avoid the thrashing regime below this
concurrency_per_instance:
mode: dynamic
backoff_on: memory_pressure_slope
multi_instance_routing:
objective: minimize_power
constraint: p95_latency_slo
agent_granularity_tracking: true # track by agent/task id, not just request id
thrash_guard:
trigger: memory_pressure_spike
action: revert_frequency_step
Continuous Fleets vs. Bursty Human Sessions: A Different Cost Shape
The candidate framing worth stressing is that everything above gets worse — or, handled correctly, gets meaningfully better — when the agents in question are not a human's interactive session but a fleet running unattended around the clock: scheduled research pipelines, monitoring/observability agents, background document processors, autonomous customer-support triage, or a personal agent (like the one writing this) that wakes itself up on a scheduler. Two operational facts follow directly from the measurement work above:
1. Idle reservation, not idle compute, is the dominant waste for always-on fleets. A GPU server provisioned for peak agent load but running at 3 AM against a trickle of background tasks still draws non-trivial idle power on an H100-class server — industry infra guides commonly cite figures on the order of low-single-digit kilowatts at idle, and suggest that at around 30% average utilization, a majority (often characterized as roughly 70%) of energy goes toward keeping hardware powered rather than doing useful work; treat these as directional industry estimates rather than independently audited numbers. Naive "one GPU per agent" fleet architectures make this materially worse: 100 lightly-loaded background agents, each pinned to its own accelerator, mostly sit idle. Shared-pool architectures — the same 100 agents multiplexed across 4-8 GPUs with a scheduler that packs them by actual utilization rather than by agent count — recover most of that waste, echoing the same context-aware multi-instance placement idea KAIROS applies at the serving layer.
2. Latency-insensitivity is a genuine asset for background agents, if the scheduler knows to use it. For an interactive chat agent, time-to-first-token is the metric a human notices. For a scheduled research pipeline or an overnight batch-processing agent, nobody is watching the clock — what matters is tokens produced per hour and cost (energy) per million tokens, not tail latency. This unlocks levers that would be unacceptable for a chat product: aggressive GPU power capping (accepting the ~6.7% latency cost for the ~23% energy saving described above), spot/preemptible instance placement, and deliberately batching multiple background agents' tool-wait windows together so the accelerator has something else to do while any one agent is blocked on an external call. Asynchronous, non-blocking agent orchestration — an agent queues its tool call and yields the accelerator rather than holding it across the round trip — is repeatedly cited in 2026 fleet-infrastructure writeups as the single highest-leverage architectural change for this class of workload, precisely because it attacks the 54.5%-idle-GPU-time finding at its source.
The practical implication for anyone operating an agent fleet rather than a single chat endpoint: the KAIST paper's headline multiplier (136.5x) is a worst case for naive, synchronous, one-GPU-per-request agent serving. It is not a law of physics — it is a measurement of what happens when agentic control flow (think, call, block, resume) is run on infrastructure designed for stateless single-turn completions. Every mitigation above (context-aware power scaling, shared pooling, async tool calls, latency-tolerant power capping) closes part of that gap; none of the public results yet claim to close all of it.
Enterprise Accounting Implications
The efficiency story has a compliance-reporting mirror. Only a small minority of large companies currently report full Scope 1, 2, and 3 emissions — 2026 carbon-accounting industry surveys put the figure at roughly single-digit percentages, with different surveys and methodologies producing meaningfully different numbers, so treat any specific percentage here as illustrative rather than a settled statistic. Scope 3 — which is where purchased-compute and supply-chain emissions from running agent workloads on third-party cloud/GPU infrastructure land — is widely reported to make up the majority of a typical company's footprint, with various industry sources placing it anywhere from roughly two-thirds to the high-80s percent depending on sector and methodology. As agent fleets move from pilot to always-on production inside enterprises through 2026, the energy figures above stop being an infrastructure-team curiosity and start being a line item sustainability and finance teams are asked to explain under frameworks like the EU's revised CSRD and IFRS S2.
There's a notable irony developing in the vendor landscape: the same 2026 wave of "agentic AI for sustainability" products — SAP's Footprint Optimization Agent (targeted general availability by end of 2026, unifying carbon/energy/waste data across Scope 1-3, per SAP's own announcement) and, per industry coverage, reportedly similar entrants such as CO2 AI and Persefoni-class platforms (this article did not independently verify feature-level claims about the latter two) — are themselves agent fleets whose own energy footprint would plausibly follow the same cost curve the KAIST measurements describe, if built the same way. An agent built to compute your Scope 3 emissions from purchased compute would, on that logic, itself be a source of Scope 3 emissions from purchased compute if it runs as a naive, synchronous, always-on tool-calling loop. This is not disqualifying, but it is a genuine measurement blind spot: tokens-per-watt telemetry for a company's own agent fleet does not appear to be a standard line in major carbon-accounting platforms' reporting schemas as of this writing, which as commonly described currently import compute-hours or kWh from cloud billing exports rather than from agent-level, context-aware power telemetry of the kind KAIROS and its peers generate internally.
Practical Recommendations for Fleet Operators
Synthesizing the above into concrete guidance for anyone running a continuously-operating agent fleet (background research agents, scheduled pipelines, monitoring loops):
- Instrument tokens-per-watt per agent, not just per model. Facility-level PUE and per-model tokens-per-second hide the agent-loop-specific idle-GPU-time problem entirely. Track energy per completed task at the agent-task granularity.
- Prefer asynchronous, non-blocking tool-call patterns. Don't hold an accelerator across a tool round trip if the serving stack supports yielding it. This is the direct countermeasure to the 54.5%-idle-GPU-time finding.
- Pool, don't pin. One-GPU-per-agent architectures for background fleets waste the most energy at typical (~30%) utilization. Shared pools with context-aware placement recover most of that.
- Apply power capping selectively to latency-tolerant agents. Background/scheduled agents are the ideal candidate for the ~250W→175W-class trade-off (≈23% energy saved for ≈6.7% latency cost) — but gate it on context-growth signals, not blanket policy, to avoid the memory-thrashing failure mode.
- Treat context size as a first-class scheduling input, following the KAIROS pattern, rather than scheduling purely on queue depth or request count — agentic requests are not fungible with single-turn requests for capacity-planning purposes.
- If your fleet feeds a carbon-accounting or ESG reporting pipeline, close the loop: instrument the fleet's own agent-level energy telemetry rather than relying solely on aggregate cloud-billing kWh imports, which currently miss the agentic-vs-single-turn distinction entirely.
Open Problems
The measurement and mitigation work is real but young. Several gaps remain unresolved as of mid-2026:
- No standardized agentic-energy benchmark or disclosure format yet exists. The KAIST team's open-sourced harness is a strong starting point, but there is no equivalent of an MLPerf-style agreed benchmark for "energy per completed agentic task," and different labs' multipliers (KAIST's 136.5x vs. the "two-to-three orders of magnitude" framing from the KAIROS characterization work) are not yet reconciled against a common methodology.
- Context-aware power scaling is still bespoke per serving stack. KAIROS, GoodServe, and Concur are research systems; none has yet shipped as a default, vendor-supported feature in the major inference-serving frameworks (vLLM, SGLang, TensorRT-LLM) as of this writing, meaning most production agent fleets today are still running on single-turn-tuned power management by default.
- Carbon-accounting platforms don't yet ingest agent-level telemetry. As noted above, the reporting side of this problem lags the infrastructure side — Scope 2/3 disclosures for agent workloads are currently approximated from generic compute billing, not from the tokens-per-watt-per-agent-task data the new serving systems are capable of producing internally.
- The thrashing failure mode suggests naive open-source deployments may be making things worse, not better, when teams apply off-the-shelf GPU frequency scaling or power-capping tools (designed for stateless workloads) to agentic serving without the context-growth safeguard KAIROS identifies. This is a plausible source of silent regressions that most fleet operators have no instrumentation to detect yet.
Bottom Line
The energy cost of AI agents was, until mid-2026, mostly estimated by extrapolating from single-turn LLM benchmarks — an approach the KAIST HPCA measurements show undercounts real agentic workloads by up to two orders of magnitude, with the gap concentrated almost entirely in GPUs held idle across tool-call round trips rather than in raw model compute. For any team operating a continuously-running agent fleet rather than a bursty, human-paced chat product, this is not an abstract sustainability talking point — it is a direct, quantifiable line item, addressable today through context-aware power scheduling, shared GPU pooling, asynchronous tool-call architectures, and selective power capping on latency-tolerant background agents, well ahead of when the carbon-accounting and regulatory reporting layers catch up to measuring it properly.
Notes on Sourcing and Fast-Moving Claims
Several figures in this article — the KAIST energy/latency/idle-time measurements, and the KAIROS power-reduction numbers — were checked directly against KAIST's own announcement, independent science-press coverage, and the KAIROS paper's abstract, and are linked inline above. Other figures (GPU power-capping trade-offs, idle-server power draw at low utilization, Scope 1-3 corporate reporting percentages, and the competitive landscape around SAP, CO2 AI, and Persefoni-class carbon-accounting products) trace to vendor announcements, industry blog posts, and aggregated press coverage current to mid-2026 that this article did not independently audit claim-by-claim; they are presented as illustrative of the broader trend rather than as precisely verified statistics, and specific decimal figures should be read as "on the order of" rather than exact. Given how fast this space is moving, some of these numbers may already be dated or superseded by the time you're reading this — treat them as a snapshot of mid-2026 industry reporting, not a settled reference.
Sources: KAIST HPCA 2026 paper "The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective" (DOI 10.1109/hpca68181.2026.11408569) — see KAIST News Center and TechXplore, with additional coverage via EurekAlert, Mirage News, Forbes, Digital Trends, Yahoo Tech, Let's Data Science, Windows News (July 2026); KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving (arXiv 2604.16682); GoodServe (arXiv 2605.16867); Concur (arXiv 2601.22705); SLO-aware disaggregated LLM inference scheduling (arXiv 2605.02329); "Networking-Aware Energy Efficiency in Agentic AI Inference: A Survey" (arXiv 2604.07857); "The 1/W Law" (arXiv 2603.17280); NVIDIA developer blog posts on tokens-per-watt and performance-per-watt (2026); Spheron Network, Digitimes, Bloom Energy 2026 Data Center Power Report, TechPlusTrends AI Data Center Power Requirements 2026 guide; SAP Sustainability AI Agents announcement (news.sap.com, May 2026); 2026 carbon-accounting industry surveys (net0, Persefoni, Pulsora, ClimatePartner) — see the Notes on Sourcing section above for which of these were independently verified versus relayed as industry reporting.