Long Context Windows for AI Agents: Architecture Patterns for 1M Token Models

Executive Summary

Claude Sonnet 4.6 launched on February 17, 2026 with a 1M token context window. Gemini 1.5 Pro has had it since early 2024. The capability has arrived — but the question agent builders should be asking isn't "how big can my context get?" It's "what does this change architecturally, and what are the new failure modes?"

The honest answer: 1M context windows solve specific problems very well, introduce new cost and latency tradeoffs, and don't eliminate the need for thoughtful memory architecture. This article breaks down what changed, what didn't, and how to make the right call for your agent.

What 1M Tokens Actually Means

The thresholds aren't linear — each jump represents a qualitative shift in what's possible.

32K tokens (~25K words): You're limited to a single long document or a short conversation history. RAG is mandatory for any knowledge beyond the immediate task.

200K tokens (~150K words): A mid-size codebase, a full legal contract package, or ~100 pages of documentation. Selective "load the whole thing" strategies become viable for self-contained tasks.

1M tokens (~750K words): The full JAX codebase (~746K tokens), a year of support tickets, or the entire source of a mid-size open-source project. Google validated this with Gemini 1.5 Pro: they fed the complete JAX codebase in a single pass and had the model reason holistically across it.

Claude Sonnet 4.6 makes the practical implication explicit in its launch notes: "run code migrations and multi-file refactors with the model seeing the full codebase context." The breakthrough is single-pass reviews — legal discovery, multi-file code migrations, large literature reviews — that previously required stitching results from multiple chunked calls.

The Memory Architecture Decision Tree

The conventional wisdom was: RAG is mandatory beyond ~32K tokens. That assumption is now broken for specific use cases. But RAG is far from dead.

When you can drop RAG and load directly into context:

Self-contained, static corpora that fit within the window
Tasks requiring cross-document reasoning (detecting contradictions, multi-hop inference) where retrieval chunking would destroy the connections
Single-session analysis where persistence isn't needed

When RAG is still necessary even with 1M tokens:

Corpora larger than 1M tokens (many enterprise knowledge bases run 10M–100M tokens)
Dynamic data — news feeds, live databases, frequently updated documentation
Interactive agents where latency matters (filling a 1M context takes ~60 seconds to process)
Cost-sensitive, high-volume workloads

The Letta team makes a useful conceptual distinction: "RAG is a retrieval pattern, not a memory system." RAG handles knowledge lookup. Agent memory manages state, learned preferences, and evolving context across sessions. These are different problems. You still need both at scale.

In-Context Memory vs. Retrieval-Augmented Memory

In-Context Memory (ICM): Load the corpus directly into the context window and reason over it natively. Best for holistic tasks where chunking breaks the reasoning. Gemini 1.5 Pro demonstrated this with Kalamang — a language with fewer than 200 speakers — where the model learned to translate at the level of a human student after being given a 500-page linguistics manual loaded entirely in context. That kind of interconnected learning is not achievable with RAG.

Retrieval-Augmented Memory (RAM): Embed and index the corpus; retrieve relevant chunks at query time. Best for large or dynamic corpora, point lookups, and cost-constrained workloads. LaRA (2025 benchmark, OpenReview) found that RAG matches or outperforms long-context loading for factual Q&A — the quality advantage of "see everything" only materializes for genuinely holistic reasoning tasks.

Hybrid (the 2026 best practice): Most production systems need both. Letta's architecture uses: (a) RAG for factual retrieval, (b) a compressed observation log for agent memory, (c) the live context window for the current task. VentureBeat reports that observational memory — summarizing past conversation turns rather than accumulating raw history — cuts agent memory costs 10x and outperforms naive RAG on long-context benchmarks.

The Numbers You Need Before Deciding

Cost

Claude Sonnet 4.6 at standard context (≤200K tokens): $3/M input, $15/M output. Claude Sonnet 4.6 beyond 200K tokens: 2x input ($6/M), 1.5x output ($22.50/M).

Filling a 1M token context costs roughly $0.50–$6 in input tokens alone, before output. For batch workloads this may be acceptable. For interactive agents running thousands of queries per day, it's a budget problem.

Gemini 2.0 Flash offers up to 1M context at $0.08/M input — dramatically cheaper for high-volume ingestion use cases. The emerging production pattern is: Gemini Flash for large-corpus ingestion, Claude Sonnet for agentic planning and complex tool use.

Latency

Processing a 1M token context takes roughly 60 seconds. RAG retrieval typically takes 100–500ms. For interactive agents, there's no contest: long-context loading is a background/batch operation, not a real-time one.

Quality — the part people miss

More context doesn't mean better reasoning. Chroma Research's "Context Rot" analysis found that models "effectively utilize only 10–20% of the context" despite nominally supporting far more. The mechanism is RoPE (Rotary Position Embedding) positional decay — tokens in the middle of a long context receive systematically less attention than tokens near the start or end.

Stanford's "Lost in the Middle" finding (2023, widely replicated): accuracy drops 15–20 percentage points based purely on where the relevant information sits in the document list, not its content quality. Gemini 1.5 Pro achieves >99.7% recall in single-needle tests at 1M tokens, but drops to ~60% in multi-needle scenarios requiring synthesis across multiple positions.

Practical implication: a 1M token window doesn't give you 1M tokens of reliable reasoning surface. It gives you a tool that works well for specific tasks and degrades quietly for others.

Anti-Patterns to Avoid

1. Treating context size as a retrieval substitute. "Just dump everything in" produces context pollution — the model processes the tokens but doesn't reason over them uniformly. You pay for 1M tokens but get the reasoning quality of a well-organized 100K context.

2. Ignoring the cost multiplier. Claude Sonnet 4.6 doubles input cost beyond 200K tokens. Without monitoring, an agent fleet that fills its context on every request can spike costs silently. Set alerts on token consumption before deploying.

3. Loading dynamic data into a static context. A 500K-token knowledge base that updates daily means every query rebuilds the full context from scratch. Corpora that change frequently belong in retrieval infrastructure, not the context window.

4. Multi-agent chains passing full context. If Agent A fills 1M tokens and passes that to Agent B, each agent in the chain re-processes the full context. The fix: pass only the compressed output — a structured summary, a tool result, a diff — keeping each agent's context lean.

5. Expecting long context to fix attention drift. If you're seeing quality degradation at 100–200K tokens, expanding to 1M won't solve it — the model's attention distribution is the bottleneck, not the window size. Compression (summarizing old context, filtering irrelevant chunks) typically outperforms expansion.

When to Upgrade vs. When to Optimize

Upgrade to 1M context when:

Users report quality drops at the end of long conversations (truncation is cutting important early history)
You're doing multi-file code operations and chunking creates coherence failures
Your task requires holistic reasoning over a corpus that fits in a single window
The workflow is async or batch (latency is not a constraint)

Invest in memory architecture first when:

Context growth is primarily driven by conversation history accumulation (compress old turns, don't accumulate them)
You're already seeing context rot symptoms at 100–200K (more context won't help)
Cost is the primary constraint (RAG + compression beats long-context on cost/quality at scale for most workloads)
Your queries are point lookups, not holistic reasoning (RAG wins on factual Q&A benchmarks)

The Broader Shift

The strategic framing that's emerging in 2026 is not "RAG vs. long context." It's a layered architecture: retrieval handles structured knowledge lookup, compression manages conversation history, and the context window holds only what's needed for the current reasoning step.

Glean's emerging agent stack report puts it well: "In 2026, 'knowledge' stops meaning a vector index and starts including semantic structure, provenance, freshness, and policy-aware retrieval." The context window gets smarter about what goes into it — not larger as a substitute for architecture.

1M token windows are a powerful tool that changes what single-pass analysis can do. Used well, they eliminate chunking complexity for self-contained corpora. Used naively, they produce expensive, slow agents with context rot. The difference is knowing which problems are actually holistic reasoning tasks and which are just point lookups that scale.