Introduction
Artificial intelligence agents designed for programming can now work for hours on the same task. They explore repositories, read files, run commands, consult documentation, modify code, and validate the result before handing control back to the developer.
This autonomy creates a problem that is easy to overlook at the beginning of a session: all of this activity produces context. When an agent reads a file, runs a command, or calls a tool, the resulting information may start competing for space with the instructions and decisions that are actually necessary to complete the task.
Tools such as Claude Code, Codex, and Cursor already use different strategies to address this problem. They summarize conversations, keep memories outside the main history, retrieve information on demand, isolate tasks in other agents, and try to preserve parts of the prompt in cache.
These techniques are part of what has come to be known as context engineering: deciding what information a model should receive, when it should receive it, and how long that information should remain available.
In this article, we will look at how this process works and explore strategies that can be applied to our own agents to reduce token consumption without simply removing important information.
What actually exists inside the context window?
Language models work with a limited amount of information represented as tokens, small units used to process text and other content. The context window represents the number of tokens the model can consider during an inference.
In a simple chatbot, we can imagine something like this:
System instructions
+ previous messages
+ new user message
+ response to be generated
For an agent performing longer tasks, the situation is more complex:
System instructions
+ project rules
+ loaded memory
+ conversation history
+ consulted documents
+ search results
+ responses from external systems
+ tool calls
+ new request
+ response to be generated
Claude Code's documentation, for example, describes its context window as containing conversation history, files, command output, CLAUDE.md, automatic memory, loaded skills, and system instructions.
This means that an apparently simple operation can have an indirect cost. If we ask an agent to look for an implementation and it runs a search that returns hundreds of lines, opens five files, and runs tests with extensive output, much of that material may begin competing for space with the information that is actually needed.
When the window approaches its limit, some strategy must be used: removing older information, summarizing it, starting a new session, or retrieving information again when needed.
Context, memory, and cache are different things
Context is the information available to the model during a given inference.
Memory is a way to persist information outside the immediate conversation history so that it can be retrieved later.
Cache, on the other hand, is not semantic memory. Prompt caching makes it possible to reuse computation associated with identical or stable parts of a prompt. The tokens still remain part of the logical context, but processing them can be faster and less expensive.
A simple way to visualize the difference is:
Context = what the model knows right now
Memory = what can be retrieved later
Cache = what has already been processed and can be reused
There is also a fourth important concept: retrieval. Instead of loading all available knowledge, the agent searches an external source and inserts only the relevant excerpts.
The question then changes from “how do we put everything into the context window?” to: how do we help the agent find the right information exactly when it needs it?
Observational Memory: summarizing while the agent works
A recent approach worth examining is Observational Memory, implemented by Mastra for long-running agents.
The idea is similar to how events are recorded in a workplace. At the end of a meeting, we usually do not need to preserve every sentence that was spoken to continue a project. A record containing decisions, changes in status, and pending items is usually more useful for the next step.
In Mastra's architecture, two auxiliary agents work in the background. The Observer follows the conversation and, when the recent history reaches a certain token threshold, turns older messages into dense, timestamped observations. These observations replace part of the raw history.
When the set of observations also grows, the Reflector comes into play. It reorganizes and condenses those notes, combines related information, and removes items that are no longer relevant.
The context then has roughly three layers:
Reflections
↓
Condensed observations
↓
Recent messages in their original form
This creates an interesting difference compared with compaction performed only when the context window is nearly full. Memory is built continuously as the agent works, preserving important decisions and changes before the history has to be discarded all at once.
There is also an effect on prompt caching. Because observations are added progressively and recent messages remain at the end, much of the prefix can remain stable across calls. According to tests published by Mastra, the implementation achieved 94.87% accuracy on LongMemEval with GPT-5 mini while maintaining a stable context window. This figure comes from a benchmark published by the authors themselves and should be interpreted within that methodology.
For agents that work for hours, this approach is particularly interesting because it combines two concerns that are usually treated separately: preserving long-term memory and keeping the context predictable enough to benefit from caching.
How Claude Code, Codex, and Cursor handle the problem
Claude Code
Each Claude Code session starts with a new context window. To carry persistent knowledge across sessions, the tool provides mechanisms such as CLAUDE.md and Auto Memory.
CLAUDE.md contains instructions written by the developer: conventions, important commands, architectural rules, and information the agent should know. The documentation recommends keeping it concise; files longer than roughly 200 lines consume more context and can reduce instruction adherence. Rules for specific paths and skills can be used to load some instructions only when they are needed.
Automatic memory follows an interesting strategy. Claude Code keeps a concise MEMORY.md file and loads only the first 200 lines or 25 KB at the start of a session, whichever comes first. More detailed information can remain in topic-specific files and be read on demand.
As the session grows, Claude Code can first clear older tool results and then compact the conversation. /compact replaces the history with a structured summary; persistent root-level instructions and automatic memory are then injected again.
Subagents also help. They receive separate context windows, allowing them to search many files and return only a summary. Delegating work, therefore, also acts as a form of context isolation.
Codex
In January 2026, OpenAI described how the Codex agent loop accumulates messages and tool results. Previous history is preserved as the prefix of the next prompt whenever possible, which favors prompt caching.
When context grows beyond a certain limit, Codex uses compaction. The Responses API provides a specific mechanism that can replace history with a smaller representation and allow work to continue.
OpenAI also reached another useful conclusion when designing large repositories for Codex: it is better to give the agent a map rather than a thousand-page manual. Instead of turning AGENTS.md into an encyclopedia, a small file can point to structured documentation that the agent consults as the task requires.
Cursor
Cursor refers to a similar approach as dynamic context discovery: providing less information up front and allowing the agent to find the rest while it works.
Very large tool results can be stored in files. Instead of placing thousands of lines directly into the conversation, the agent receives a reference and reads only the sections it needs.
The same principle can be applied to MCP tools. In an A/B test published by Cursor, loading tools dynamically reduced the agent's total token usage by 46.9% in runs that used MCP, with the exact impact varying according to the number of installed servers.
Cursor also summarizes long conversations and uses smart condensation for large files, initially presenting structural elements such as classes, methods, and signatures.
Despite the differences, the pattern is similar:
keep little permanent context
↓
discover information on demand
↓
compact history when necessary
↓
preserve important knowledge outside the conversation
The best context may be the context that never entered the conversation
Consider an agent responsible for analyzing contracts and answering a question about a specific clause.
An inefficient strategy would be to open complete documents, consult attachments one after another, and keep all of that content in the conversation until the answer is found.
Question
↓
several complete documents
↓
attachments and intermediate results
↓
relevant information
An alternative is to first search for the parts of the knowledge base that are most likely to answer the question:
Question
↓
Search
↓
Knowledge index
↓
Most relevant excerpts
↓
Agent
The same principle can be applied to internal policies, customer support knowledge bases, or operational documentation. The information remains available, but it enters the context window only when there is a reason to use it.
This is one of the most natural uses of RAG (Retrieval-Augmented Generation) in agents.
Semantic search, RAG, and hybrid retrieval
Traditional search works very well when a known term is available. A query for a contract number, product name, or customer identifier can quickly locate a record.
In other cases, the question describes an intent. A user may want to know which conditions allow a service to be canceled early without knowing the exact wording used in the documents.
This is where embeddings come in: numerical representations that make it possible to compare semantic similarity. Documents can be divided into smaller chunks and stored in an index.
When the agent asks a question, the system searches for the semantically closest chunks and returns only a small number of results.
More robust solutions can combine lexical and semantic search. Algorithms such as BM25 help rank documents by textual relevance, while embeddings help when the wording in the question differs from the wording in the source.
For business agents, this architecture makes it possible to consult large knowledge bases without loading the entire body of material for every request.
A technical example: codesearch
The same principle is clearly visible in software development. One interesting example is codesearch, a semantic search tool created specifically for agents. It runs locally as an MCP server and combines vector search with BM25, symbol navigation, and structural code splitting.
One relevant detail for context management is how its searches work: codesearch can return metadata about results first and leave the full content for a later call. The agent discovers where the information is likely to be before bringing the code into its context window.
The tool illustrates a principle that can be transferred to other domains: locate first, load second. A financial agent can identify the relevant records before retrieving their details; a customer support agent can identify the appropriate procedure before loading the complete document.
Another example: Graphify and graph search
A complementary approach appears in Graphify, a tool for creating queryable knowledge graphs. It can process code, documents, PDFs, and other content, turning concepts and relationships into a persistent structure that can be queried later.
Instead of rereading the complete source for every question, the agent can query the graph to discover which entities are related, find paths between concepts, or retrieve only a subgraph associated with the current question. The tool provides operations such as query, path, and explain, and can expose the graph through MCP for structured access by agents.
Documents and other sources
↓
Graphify
↓
Persistent knowledge graph
↓
specific query
↓
subgraph relevant to the agent
One useful aspect for context management is the persistence of graph.json. Once the structure has been built, later queries can use the graph without requiring every original file to be read again.
Graphify also records the origin of relationships. Connections can be marked as EXTRACTED, when they appear explicitly in the sources, or INFERRED, when they were derived during processing. This distinction helps the agent treat inferred relationships with greater caution.
For graph search, this approach is interesting because it can answer questions about the structure of knowledge. In a business knowledge base, for example, it may be more useful to discover how a policy connects to a process than to locate documents that are only semantically similar.
codesearch and Graphify therefore represent two complementary ways to reduce context. The first helps locate relevant excerpts before loading them. The second makes it possible to explore already-structured connections and retrieve only the portion of the graph needed for the current question.
When similarity is not enough: relationships and graphs
Semantic search is effective at finding information related to a question. Some tasks, however, depend on relationships between entities.
Consider an agent analyzing suppliers. Finding documents that mention a company is different from discovering which contracts are associated with it, which business units depend on those contracts, and which processes would be affected by a change.
Those relationships can be represented as a graph:
Supplier
↓ has
Contract
↓ serves
Business unit
↓ runs
Process
The nodes represent entities, and the connections record their relationships.
This approach makes it possible to combine semantic retrieval with structural knowledge. GraphRAG, for example, uses graphs to organize entities and relationships before retrieval.
In software development, work such as GraphCoder applies a similar idea to relationships within a repository.
Graphs are most useful when structural relationships justify the additional cost of creating and maintaining the index. For many business knowledge bases, textual and semantic search already cover a large share of queries.
Tools can also produce unnecessary context
Modern agents use tools to query external systems, search databases, or perform operations. Every response from those tools can enter the context.
Imagine an operations agent querying a system to check delayed orders. If the tool returns hundreds of fields for each order when only status, date, and owner are needed, the model receives a large amount of information that contributes little to the decision.
A good agent interface should therefore consider how much context each tool produces.
Agent
↓
Tool
↓
Complete system response
↓
Filtering
↓
Relevant information
↓
Context
In software development, RTK (Rust Token Killer) is a specific example of this principle. The tool reformats command output into more compact versions before it reaches the agent.
The same logic can be applied to business integrations. APIs and tools built for agents can provide summarized responses, pagination, and filters suited to the task.
There is, however, a balance. An overly compressed response may force the agent to make a second query to recover details that were removed.
For that reason, evaluation should consider tokens, cost, and quality per completed task, rather than only the size of an individual response.
What happens when context inevitably becomes large?
Even with retrieval and efficient tools, long-running tasks accumulate history. The agent has made decisions, modified files, received corrections, run tests, and discovered important information.
At some point, keeping everything becomes impractical.
One solution is compaction:
100,000 tokens of task history
↓
compaction
↓
Current objective
Important decisions
Relevant files
Changes made
Tests performed
Known issues
Next steps
↓
much smaller context
Claude Code, Codex, and Cursor all implement some form of this process.
In Claude Code, /compact replaces the history with a summary, while persistent mechanisms such as the root CLAUDE.md and Auto Memory are injected again.
Cursor also summarizes long conversations. In its dynamic discovery strategy, the history can remain available for retrieval: if an important detail disappears from the summary, the agent can search the earlier material.
Codex uses the Responses API compaction feature and can do so automatically when the history passes a certain limit.
Every summarization process involves choosing which information will be preserved, which creates the possibility of context loss. If a conversation contains twenty decisions and the summary preserves nineteen, the discarded one may be exactly what is needed several hours later.
Handoff: starting a new session without starting over
A simple technique I have been using is to ask the agent to create a handoff document before ending a long session.
# Objective
# Current status
# Important files
# Decisions made
# Changes made
# What has already been tried
# Known issues
# Next steps
A clean session can then be started and the new agent can be asked to read that document.
Instead of loading 120,000 tokens of task history, the new session can receive:
project instructions
+ HANDOFF.md
+ files needed for the next step
This works as a form of explicit, reviewable compaction. We can ensure that critical decisions were recorded and separate temporary information from the information that needs to survive.
The same approach can also transfer work between different agents. One agent can investigate the problem, produce the handoff, and allow another to continue the implementation without receiving the entire previous history.
Fewer tokens do not always mean lower cost
So far, the focus has mainly been on reducing context size. There is another important optimization, however: prompt caching.
Consider several consecutive inferences:
Call 1
[20k stable tokens][2k new]
Call 2
[20k stable tokens][2k][1k new]
Call 3
[20k stable tokens][2k][1k][800 new]
Much of the beginning remains unchanged. With prompt caching, the provider can reuse processing for a prefix that has already appeared:
[ cached prefix ][new content]
This is why prompt organization matters.
OpenAI explains that cache hits depend on prefix matches. Static content, such as instructions, should remain at the beginning whenever possible, while variable information should be appended later.
In Codex, preserving that prefix is an explicit concern of the harness. Changing the model, the tool list, or certain settings can cause a cache miss. OpenAI has even described an issue in which MCP tools were enumerated in a nondeterministic order, reducing cache reuse.
Providers may charge different rates for input, cache write, and cache read. Two agents processing roughly the same logical amount of context can therefore have different costs depending on how effectively they reuse prefixes.
There is one important distinction:
Prompt caching can reduce cost and processing, but it does not make tokens disappear from the context window.
This means caching and compaction solve different problems.
Compact the context or preserve the cache?
This difference creates an interesting trade-off.
Imagine an agent with a large history whose prefix is being reused efficiently. Removing or changing content in the middle of that history may free space in the context window, but it can also alter the prefix used by the cache.
In other words, one optimization may interfere with another.
For that reason, context engineering should not be reduced to “using fewer tokens.” At least four factors need to be balanced:
response quality
↕
relevant context
↕
available space
↕
cache hit
↕
cost and latency
Sometimes it is worth keeping a larger context because much of it is being retrieved from cache. In other cases, the context has accumulated so much noise that compaction or a new session produces better results even if part of the cache has to be rebuilt.
A practical strategy for long-running agents
All of these techniques may seem independent, but they work better as layers.
A practical strategy would be:
- Keep permanent instructions small.
CLAUDE.md,AGENTS.md, Rules, and similar files should function mainly as maps and truly universal rules. - Keep detailed documentation outside the initial context. The agent should know where to find it.
- Search code before reading large numbers of files. Textual, semantic, or hybrid search can reduce unnecessary exploration.
- Load tools on demand whenever possible. Large tool schemas also consume context.
- Control noisy output. Logs and commands should return the amount of information needed without hiding critical data.
- Use subagents for isolated investigations. Research can happen in another context window and return only its conclusion.
- Persist important learning in external memory. Do not depend on an old conversation to retain rules that will be needed tomorrow.
- Compact at natural points in the task. A transition between stages is a better moment to summarize than waiting for the context window to become completely full.
- Create handoffs for long-running tasks. A new session with well-documented state can be more efficient than a conversation that grows indefinitely.
- Preserve stable prefixes when this does not conflict with quality. A high cache-hit rate can significantly reduce the cost of repeated calls.
The strategy can be summarized as a pyramid:
Compaction
when needed
▲
Handoff / memory
▲
Retrieval on demand
▲
Concise tool output
▲
Small, stable permanent instructions
The main point is to avoid treating the context window as a storage area where every potentially useful piece of information should be placed.
It is closer to a system's working memory.
Conclusion
Context windows continue to grow, but that does not remove the need to manage them.
An efficient agent does not need to keep everything it has ever seen available at all times. It needs to know what to keep, what to forget, where to look, and what to retrieve when needed.
Claude Code, Codex, and Cursor are converging on this principle through different approaches. Persistent memories preserve knowledge across sessions. Code search and RAG retrieve specific information. Graphs help when structural relationships matter. Compaction shortens long task histories. Handoffs make it possible to restart sessions while preserving state. Prompt caching reduces the cost of information that must remain present.
For that reason, perhaps the most important strategy for saving tokens is not finding the best way to compress everything afterward.
It is preventing unnecessary information from entering the context in the first place.
As artificial intelligence agents move from short conversations to tasks that last for hours or days, context management shifts from being only a technical limitation of the models to becoming part of the architecture of the system itself.
Sources
GRAPHIFY LABS. Graphify - queryable knowledge graphs for AI assistants. GitHub. Accessed Aug. 29, 2026.
FLUPKEDE. codesearch — Multi-repo semantic code search for AI agents. GitHub. Accessed Aug. 26, 2026.
MASTRA. Observational Memory: 95% on LongMemEval. Feb. 9, 2026. Accessed Aug. 26, 2026.
MASTRA. Anatomy of a harness: building a coding agent that can run for hours. 2026. Accessed Aug. 26, 2026.
ANTHROPIC. How Claude remembers your project. Claude Code Docs. Available at: https://code.claude.com/docs/en/memory. Accessed Aug. 26, 2026.
ANTHROPIC. Explore the context window. Claude Code Docs. Available at: https://code.claude.com/docs/en/context-window. Accessed Aug. 26, 2026.
ANTHROPIC. Manage sessions. Claude Code Docs. Available at: https://code.claude.com/docs/en/sessions. Accessed Aug. 26, 2026.
OPENAI. Unrolling the Codex agent loop. Jan. 23, 2026. Available at: https://openai.com/index/unrolling-the-codex-agent-loop/. Accessed Aug. 26, 2026.
OPENAI. Harness engineering: leveraging Codex in an agent-first world. 2026. Available at: https://openai.com/index/harness-engineering/. Accessed Aug. 26, 2026.
CURSOR. Dynamic context discovery. Jan. 6, 2026. Available at: https://cursor.com/blog/dynamic-context-discovery. Accessed Aug. 26, 2026.
CURSOR. Summarization. Cursor Docs. Available at: https://docs.cursor.com/en/agent/chat/summarization. Accessed Aug. 26, 2026.
CURSOR. Memories. Cursor Docs. Available at: https://docs.cursor.com/en/context/memories. Accessed Aug. 26, 2026.
RTK. RTK — Rust Token Killer. GitHub. Available at: https://github.com/rtk-ai/rtk. Accessed Aug. 26, 2026.
MINISHLAB. Semble — Semantic Code Search. GitHub. Available at: https://github.com/MinishLab/semble. Accessed Aug. 26, 2026.
LIU, et al. GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Model. arXiv, 2024. Available at: https://arxiv.org/abs/2406.07003. Accessed Aug. 26, 2026.
MICROSOFT. GraphRAG. GitHub. Available at: https://github.com/microsoft/graphrag. Accessed Aug. 26, 2026.
