An AI agent can only think about what fits in front of it. That "in front of it" is the context window: the span of text the model reads before deciding its next move. For a quick question it never fills up. For a long-running agent that reads documents, calls tools, and works across many steps, the window fills fast, and once it is full the agent has to forget something. Managing what stays and what goes is one of the quiet skills that separates an agent that holds a long task together from one that loses the plot halfway through.

This post explains context window management in plain language: what the window is, why it constrains agents, and the three strategies, summarization, retrieval, and persistent memory, that keep an agent working with more information than the window can hold. It pairs with AI agent memory explained and the retrieval deep dive in agentic RAG vs RAG.

What the context window is

The context window is the maximum amount of text a language model can take in at once, measured in tokens, where a token is roughly three-quarters of a word. For an agent, that window is not just the user's question. It holds the system instructions, the running conversation, the descriptions of every tool, the results those tools returned, and any documents the agent pulled in. All of it competes for the same finite space.

The right way to picture the window is as working memory, not long-term memory. It is the desk the agent works on, not the filing cabinet behind it. A desk can only hold so many papers before something falls off the edge. Modern models have large desks, but agents fill them quickly because every tool call, every observation, and every reasoning step adds more paper. The skill is keeping the right papers on the desk.

Why it limits agents

Three pressures make the window a real constraint rather than a footnote. The first is overflow. A multi-step agent accumulates history with every loop, and a long enough task will exceed any window. When that happens, something has to be cut, and cutting the wrong thing makes the agent forget a key instruction or an earlier result. The state-management side of this is covered in AI agent state management.

The second pressure is cost. Models charge per token, so every token in the window is money, and a bloated context makes each step more expensive; the arithmetic is in AI agent cost models explained. The third is accuracy. Research on long-context models, notably the "lost in the middle" study by Liu and colleagues, found that models retrieve information best when it sits at the start or end of a long context and can miss facts buried in the middle. So a stuffed window is not just costly; it can quietly degrade the agent's reasoning. These three pressures are why management beats simply making the window bigger.

Summarization and compaction

The most direct way to fit a long history into a small window is to compress it. Summarization replaces a long run of raw messages and tool outputs with a short recap that keeps the important facts and drops the noise. Context compaction is the same idea applied on a schedule: every so often, the agent folds its recent history into a summary and continues from there, so the window never overflows.

The trade-off in compaction

Compaction buys continuity at the price of detail. A summary is lossy by definition, so a fact that seemed unimportant when it was compressed is gone if it turns out to matter later. The craft is in deciding what survives the squeeze: goals, decisions, and unresolved questions should persist, while chatter and redundant tool output can go. A good compaction step is itself a small reasoning task, which is why a sloppy summarizer can hurt an agent as much as overflow would.

Retrieval

The second strategy stops trying to hold everything in the window at all. Retrieval, often called retrieval-augmented generation or RAG, keeps information in an external store and pulls in only the pieces relevant to the current step. Instead of loading a hundred-page policy into context, the agent searches the policy and inserts the two paragraphs that answer the question at hand. The window stays small while the agent has access to a large body of knowledge.

When retrieval is the right tool

Retrieval shines when the agent needs to draw on far more reference material than any window could hold: a documentation set, a knowledge base, a history of past tickets. It also keeps that material current, since updating the store updates what the agent can find without touching the agent itself. The cost is a new moving part, the retrieval system, which has to surface the right passages; retrieve the wrong ones and the agent reasons over irrelevant text. The difference between plain retrieval and an agent that decides what to fetch is the subject of the agentic RAG vs RAG comparison.

Persistent memory

The third strategy spans runs rather than steps. Persistent memory is durable storage of facts the agent should carry across sessions: a user's preferences, decisions made last week, the state of an ongoing project. Where the context window is the desk and retrieval is the filing cabinet, persistent memory is the notebook the agent keeps between days. Without it, every run starts from zero and the agent feels amnesiac.

What belongs in memory

The discipline with memory is selectivity. Save the durable and the decision-shaping, a customer's standing instructions, the outcome of a prior task, a correction the user made, and leave the transient out. A memory that records everything becomes its own retrieval problem, since the agent then has to search its memory to find what matters. The deeper treatment of how agents store and recall across time is in AI agent memory explained, which sits alongside this post in the same cluster.

Putting it together

Real agents do not pick one strategy; they layer all three. The window holds the live task. Compaction keeps the running history from overflowing. Retrieval brings in reference material on demand. Persistent memory carries the durable facts across runs. Each handles a different timescale, so together they let an agent work on a long, knowledge-heavy task without ever needing the whole world in front of it at once.

How we tune context at Gravity

Building Gravity's reference agents, the lesson that stuck was that more context is not the same as better context. Our early instinct was to give the agent everything and trust the large window to sort it out. Accuracy actually improved when we did the opposite: keep the window lean, compact aggressively, and retrieve narrowly, so the agent reasoned over a small set of highly relevant facts rather than a huge pile of mostly-irrelevant ones. Curating the context turned out to matter more than expanding it, which is also why a bigger model window is a convenience rather than a fix.

What this means for buyers

If you run agents rather than build them, context management is invisible, and that is the point: it is the builder's job to keep the agent coherent over long tasks. You feel it only as reliability. An agent that forgets your earlier instruction halfway through a job is usually mismanaging its context, while one that holds the thread across a long task is doing this work well. On a marketplace you describe the outcome and the builder handles the window, but it helps to know that an agent losing track is a solvable engineering problem, not an inherent flaw. The wider view of agent capability is in what can an AI agent actually do.

Frequently asked questions

What is a context window in an AI agent?

The context window is the amount of text a language model can consider at once, measured in tokens. For an agent, it holds the instructions, the conversation so far, tool results, and retrieved data. Everything the agent reasons over has to fit inside this window, so it is a hard working-memory limit.

Why is context window management important for agents?

Long-running agents accumulate history fast, and a full window forces the agent to drop information or fail. Managing context keeps the relevant facts in view, controls cost since every token is paid for, and avoids the accuracy drop models show when key details are buried in a very long window.

How do agents handle more information than fits in the context window?

They use three main strategies. Summarization compresses old history into a short recap. Retrieval, often called RAG, stores information externally and pulls in only the relevant pieces when needed. Persistent memory saves durable facts across runs. Most agents combine all three to stay within the window.

Does a bigger context window solve the problem?

Not fully. Larger windows help, but cost rises with every token, latency grows, and research shows models can miss information buried in the middle of a long window. A big window is a tool, not a cure. Good agents still curate what goes in rather than dumping everything.

What is context compaction?

Context compaction is periodically replacing a long, raw history with a shorter summary that keeps the important facts and drops the noise. It lets a long-running agent continue without overflowing its window, trading some fine detail for the ability to keep working coherently over many steps.

Three takeaways before you close this tab

Sources