On Contextual Engineering and Efficient Agents Part 1
From prompt engineering to contextual engineering: treating LLM context like a database instead of a notepad.
The promise of expanding context windows has been one of the most exciting developments in large language models. We can now feed entire documents, codebases, and conversations into our models without hitting the dreaded token limits that plagued earlier generations. But here's the thing: just because we can stuff more tokens into our context doesn't mean we're using them effectively.
The benchmarks look great. Models can find needles in haystacks, extracting specific sentences from novels with impressive accuracy. But in practice, when you're building real applications, you quickly discover that finding information is only half the battle. The real challenge is synthesizing and aggregating that information in meaningful ways, and current approaches still struggle with this fundamental problem.
Consider a common scenario: you're building an agent workflow to generate a full-length novel. Your system needs to pass chapters to a critic tool for editing and refinement. In today's naive implementations, the agent would dutifully transcribe the entire chapter into a tool call parameter. If that chapter is 10,000 words, your model sits there generating every single word, even though that exact text already exists somewhere in your context. You don't just lose efficiency here, there is a real chance of transcription error.
The Problem with Current Context Management
A core issue lies in how we think about memory and context in agent workflows. Most implementations treat context as a monolithic blob that either gets included or excluded entirely. There's no nuanced decision-making about what information deserves active attention versus what can be archived or referenced.
When I think about agent architectures, I see two main components: the language model itself that generates text and follows instructions, and the orchestrator - the wrapper code that handles pre-processing, post-processing, and tool execution. Currently, most of the intelligence around context management sits with the model, but we have an opportunities to make the orchestrator smarter about these decisions.
The solution I've been exploring involves two key ideas: dynamically choosing which information to keep active versus archive, and being more efficient in how we leverage the context we do decide to use. Both approaches borrow heavily from lessons learned in programming languages and text editors.
Side note: Maybe as a side note to this topic, it seems to me that many of the optimizations we are currently seeing with LLMs and agentic workflows are around context engineering, which I internalize as system design for agents. We care about data structures and architectures that improve efficiency and consistency over pure compute. And from this I think we are seeing lots of "inspiration" from how these fields developed with hardware, software, and cloud technologies.
Dynamic Context Selection
The first piece involves making context management more intelligent and dynamic. Instead of keeping everything in active memory, the orchestrator should make informed decisions about what context deserves immediate attention based on the current intent and workflow state.
This includes handling different types of content differently. Reasoning tags that models sometimes generate might be valuable for a few turns but become noise after that. Response messages might need longer retention. RAG retrievals might have their own lifecycle based on relevance and freshness.
The key insight is that these decisions shouldn't be static rules baked into the system. They should be dynamic, contextual choices made by the orchestrator based on the current state of the conversation and the specific requirements of the task at hand.
Reference-Based Context Management
The second piece involves treating context more like a programming language treats data structures. Instead of duplicating text generation unnecessarily, we can work with context by reference and tokenize it to make edits and searches more efficient.
Here's how this works in practice: every output from the model gets wrapped in XML tags that include metadata like IDs, descriptions, and attributes. A response might look like this:
HTML
<response id="123654ABY" ttl="10" description="hotel recommendation">I recommend the Grand Hotel in downtown Paris for your business trip.</response>
These tags serve multiple purposes. They allow the model and orchestrator to better categorize different types of content. They provide hooks for external processes to make decisions about retention and retrieval. And they enable reference-based operations that dramatically improve efficiency.
The time-to-live (TTL) attribute is particularly powerful. Different types of content can have different lifespans in active context. A thinking tag might only be relevant for 2 turns, while a RAG retrieval might stay active for 10. Some content might even move to "cold storage" where it's still accessible but not actively consuming context space.
Tokenized Context Operations
For longer context operations, I've been experimenting with tokenizing content at the word level as JSON objects. This might sound like overkill, but it enables some powerful optimizations.
Consider the sentence "Find me a good hotel for my trip to Paris." In a tokenized system, each word becomes a structured object with its own ID and attributes. When you need to make an edit, to change the destination from Paris to Germany, the model doesn't need to regenerate the entire sentence. It can simply output an edit operation: replace the token with ID "xyz123" (Paris) with "Germany."
This approach reduces both generation time and transcription errors. Instead of generating dozens of tokens, the model outputs a concise edit instruction. The orchestrator handles the actual text manipulation, ensuring accuracy and consistency.
Going back to the novel writing example, when you need to pass a chapter to an editing tool, you don't pass the raw text. Instead, you pass a reference: "process tokens from ID abc123 to ID def456." The orchestrator resolves these references and constructs the appropriate tool call. The model never has to regenerate content that already exists in context.
Python
tokenized_string = TokenEditor("Hello world!")>>> {"tokens": {"BAli9": {"value": "Hello"},"I0PY9": {"value": " "},"88krG": {"value": "world"},"4qSzY": {"value": "!"}},"order": ["BAli9","I0PY9","88krG","4qSzY"]}editor.insert(" beautiful", 'BAli9')
Tool Parameter Categories
This approach becomes especially valuable when you consider the different categories of tool parameters in modern agent workflows. Traditional tools work with simple parameters—integers, booleans, short strings. These are straightforward and don't benefit much from optimization.
Semantic tools represent a middle ground. These might involve parameters like "find me a good hotel for my trip to Paris." They are manageable strings that get passed to other language models for processing. The current orchestrator-based approaches handle these reasonably well.
But long context tools are where the real challenges lie. These involve multiple instructions, paragraphs of context, or entire documents as parameters. When you need to call a tool with a full document, naive approaches break down quickly. This is exactly where reference-based context management shines.
Implementation Considerations
The obvious objection to this approach is complexity. By tokenizing and structuring context, are we making the input so complex that model performance actually degrades? This is a valid concern that requires careful experimentation.
My hypothesis is that the benefits outweigh the costs, especially for longer, more complex workflows. The reduction in generation time, the elimination of transcription errors, and the improved context utilization should more than compensate for any additional complexity in the input format.
But this needs to be validated through systematic testing across different model architectures, context lengths, and task types. The sweet spot likely varies depending on the specific use case and the sophistication of the underlying model.
Performance Characteristics
It should be noted (and I will probably make a separate post about this) that in some initial testing of the tokenization, I noticed that using tools to edit long text string resulted in less tokens used in output (generally the more expensive of the two) as well as some speed increases, but this was not universal. Depending on how you tokenize the content (and I tried more than a few approaches like w/wo space tokenization or w/wo the order structure being explicit) I experiences some cases where the tokens into the model were so large that it countered any performance gains from the reduced tokens out.
Again, this was limited testing that I need to do a bit more work on. The chief tradeoff seems to be:
- Many On-Demand LLMs have a pricing structure where tokens out are ~5-10x as expensive as tokens in.
- LLMs generally can accommodate increases in input tokens better than output tokens due to processing inputs in parallel.
So the statements is: "Can we structure our tokenization in such a way, where we have decreased the output tokens to sufficiently improve performance even with the increased input tokens?"
I also tested accuracy in the tasks given and saw (on average) increased performance when using the tokenization, but this is definitely a separate post.
Looking Forward
This exploration into contextual engineering represents a natural evolution from prompt engineering toward what some call "software 3.0." It's the integration of traditional programming concepts with language model workflows. We're essentially creating data structures and processes specifically designed for LLM-based systems.
As context windows continue to expand and agent workflows become more sophisticated, we'll need increasingly nuanced approaches to context management. The key insight is that intelligence shouldn't reside solely in the language model—the orchestrator has an important role to play in making these systems efficient and effective.
References
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article