Context Engineering Is Not About Adding More Context

An LLM feature rarely collapses under context on its first day.

Version one is lean: a system prompt, a user request, and one model call. After the first wrong answer, the prompt gains another rule. The application then adds retrieval for private documents, tools for live data, conversation history for continuity, and task logs for recovery.

Each addition makes sense on its own. A request captured a few months later may contain instructions, documents, search results, tool output, chat history, user preferences, failure records, and an old summary. The model now sits at a desk covered in paper. Everything is within reach, but the page that matters is harder to find.

Saying “the model has a large context window” does not solve that problem.

Chinese version of this article

From One Model Call to an Agent maps prompts, context, retrieval-augmented generation (RAG), memory, tools, and agents. This article takes one part of that map further. It examines what an application should put into a model call and what it should leave outside.

A larger window gives capacity, not quality

A context window sets the maximum number of tokens available to one inference. It works like the surface area of a desk. A larger desk holds more material, but it does not organize that material or make the right page easier to notice.

Long context creates four distinct problems.

The first is cost and latency. Longer input increases transfer size and the model’s prefill work before it generates the first output token. In an agent loop, later calls may resend material from every earlier step, so one extra block of context can incur cost several times.

The second is interference. If only two of ten retrieved documents support the answer, the other eight are not free background. They compete with the useful evidence. Stale rules, duplicate summaries, and conflicting records make the problem worse because the model must infer which source deserves trust.

The third is position. Research such as Lost in the Middle shows that models can use information unevenly across a long input. A fact’s presence in the context does not guarantee reliable use. Burying a critical constraint between thousands of tokens and ending with “follow all instructions above” expresses hope, not control.

The fourth problem is output space. The context window must also leave room for the answer. When input consumes the budget, the model cannot finish a report, code patch, or JSON object. Some APIs reject the call, while some runtimes truncate earlier messages. Neither outcome belongs in a production strategy.

A larger window solves “this material does not fit.” Context engineering addresses different questions: what belongs, where it belongs, and when it should leave.

What occupies one model call

Context includes more than the latest user message. A tool-using application may assemble all of these sources for one call:

• System instructions and safety rules
• The current task and the user’s latest constraints
• Evidence retrieved from documents, databases, or search
• Results from recent tool calls
• Recent conversation and task state
• A small set of long-term memories
• An output schema or format example

The application must reserve space for the model’s output as well.

These sources do not have equal status. Instructions stay stable. The current task must remain intact. Retrieval evidence can compete on relevance, while old tool logs may only need a compact conclusion. Conversation history loses value as the task changes.

Concatenating everything into one string works until the budget runs out. It also hides which source caused the failure. Separate sections make it possible to assign each source a budget and an eviction policy.

A single global rule such as “drop the oldest message when over budget” is too blunt. The oldest message may contain the task goal. I prefer to reserve output space first, protect instructions and the current task, let evidence and tool results share a flexible area, and retain only the state needed to continue.

That policy is conservative. It is still better than receiving half a JSON object because the input took the model’s last available tokens.

Context engineering has five operations

Context engineering extends beyond prompt wording. Prompt engineering asks how to express the task. Context engineering decides where supporting material comes from, how it enters a call, and how it leaves as the task grows.

The implementation usually contains five operations.

Selection. Find candidate material for the current question, then choose what enters the working set. A shipping request does not need the full refund policy. A database timeout investigation does not need every frontend log.

Compression. Reduce long source material while preserving facts, provenance, and time. A graceful paragraph that loses the error code, amount, version, or file path is not a useful summary. The missing detail may be impossible to recover later.

Ordering. Keep stable rules in a stable location, mark the current task, and place important evidence near the instruction that uses it. When sources differ in authority or age, label those differences instead of asking the model to guess.

Isolation. Give each subtask its own working set. A retrieval step may not need writing-style rules. The model composing the final answer does not need raw tool debugging logs. Isolation also reduces the amount of sensitive data exposed to each call.

Deletion. Remove expired plans, duplicate observations, and summaries superseded by newer facts. Deletion is part of normal long-task execution, not an emergency response. Memory that only accepts writes eventually becomes another log archive.

None of these operations is mysterious. The hard part is that they cannot live in a prompt template alone. Retrieval, state storage, budget calculation, and the runtime must enforce them together.

RAG, memory, and tool results need different policies

Several context sources look similar after they become text. They still require different retention rules.

RAG involves more than returning the top few text chunks. The system must decide what to search, how many results to keep, how to remove duplicates, and how to preserve source and freshness metadata. Memory has a similar distinction: storing a fact does not make the model know it. The application must retrieve that fact when it becomes relevant.

Tool results create the fastest context growth. A log tool may return hundreds of lines. The model reads them and calls another tool, while the runtime appends both raw results and both explanations to the conversation.

After three or four steps, the original task may occupy a small corner of the input. A safer policy keeps the latest observations in detail and compresses older results into a checkpoint with evidence pointers.

Long tasks get hard on the second call

You can assemble context for one call by hand. The second call introduces the real design work: how much of the first answer stays, whether tool output returns verbatim, whether failed attempts matter, and how new evidence overrides old conclusions.

Assume each step adds an observation of similar size and every later call resends the full history. Call ten pays for the first ten observations, not only the tenth. Across the task, cumulative input grows approximately with the square of the step count.

An unbounded loop therefore expands the context window, latency, and bill together. A long task needs a context lifecycle, not a chat transcript that grows forever.

The most recent steps can retain detail. Older steps become a checkpoint. A useful checkpoint records the goal, confirmed facts, completed actions, failed attempts, and missing information.

That record must support recovery after an interruption. “The task made good progress” cannot restart anything.

Once a subtask finishes, its process material can leave the parent context. Only its conclusion, evidence, and unresolved questions return. A later stage retrieves source material again when it needs the details.

This approach performs more assembly work between calls. In return, each model invocation receives a bounded workspace with a clear purpose.

A practical context assembly step

You do not need a large agent framework to establish these boundaries. Separate material collection from the model call first.

const candidates = await collectCandidates(task, state)
const evidence = selectAndRank(candidates, task)
const recent = keepRecentObservations(state.events, 2)
const checkpoint = compactOlderEvents(state.events)

const context = assemble({
  instructions: stableInstructions,
  task: normalizeTask(task),
  evidence: fitToBudget(evidence, budgets.evidence),
  recent,
  checkpoint,
  outputSchema,
  reserveForOutput: budgets.output,
})

const result = await model.generate(context)
await persistResultAndState(result, state)

The important part is the boundary between each function. Candidate material is not the final context. Recent observations and older state use different policies. Output space affects assembly before the call, and the runtime persists state outside the context after the call.

A mature implementation can attach metadata to each block: source, timestamp, priority, ttl, and sensitivity. Provenance, expiry, and permission should not exist only as prose inside the material.

Measuring whether context engineering works

Lower token usage is one metric, not the result. Removing too much context can produce a faster wrong answer.

I track three groups of measurements.

The first covers cost and latency: input tokens per call, cumulative tokens per task, time to first token, total duration, and the p50 and p95 distributions. An average can hide a small set of requests that carry most of the context.

The second covers task quality. Use a fixed set of real requests and verify the evidence behind each answer. For structured output, run schema and domain validation. For tools, inspect the retrieval path as well as the final prose.

The third covers runtime behavior: candidate count, selected count, truncation by section, checkpoint reuse, and stop reason. Without these records, context failures collapse into “the model is unstable,” which does not identify a fix.

An evaluation set does not need hundreds of cases at the start. Twenty or thirty tasks that previously failed can expose useful differences. Fix the input, required evidence, and pass condition before comparing a new model or assembly policy.

Failure patterns worth watching

The most common mistake is treating “the model may need this later” as “include this on every call.” Potentially useful material belongs in retrievable storage. It enters context when the current step needs it.

Premature summarization creates another failure. The system saves tokens by replacing source text with a smooth paragraph, then discovers that the paragraph omitted the number that decides the case. Keep source pointers in summaries and retain original excerpts for high-risk facts.

Applications also give every source equal trust. A live database query, a three-month-old chat summary, a new user statement, and the model’s previous guess should not share one level. Record provenance, freshness, and confidence explicitly.

Finally, some runtimes wait for an overflow error and then remove the oldest messages. The API returns 200, but the task goal may have disappeared. A controlled degradation policy works by section: deduplicate, drop low-relevance evidence, compact older observations, and protect the task and stable rules until the end.

The Novevia implementation

I met this problem in Novevia, an AI fiction-writing application. Its first whole-book review feature compressed the setting, chapter cards, facts, and memories into one large request. As a book grew, the first request grew with it. Two model calls once took more than eight minutes.

I stopped trying to write a stronger giant prompt and changed context assembly instead. The first call now receives a manifest. The model retrieves chapters, story routes, or facts when it needs them. Each call has a separate budget, older tool results get compacted, and task state remains in the database until a later step needs it.

The design also forms a bounded ReAct-style loop, but ReAct did not remove the timeout by itself. The smaller working set, timed retrieval, context eviction, and stop conditions did the work.

I Stopped Sending the Whole Book to the Model describes the read-only tools, step limits, validation, recovery, and rollback behind that implementation. Novevia supplies one concrete case. The assembly problem appears anywhere an LLM task lasts beyond one call.

Too little context forces the model to guess. Moving the entire archive into the window creates a different failure.

A practical target is narrower: give each call the material required for its current step, without enough unrelated material to obstruct the work. Context engineering starts there.