What If Your AI Didn't Need to Read the Entire Document at Once?
I used to think analyzing huge documents meant either fitting everything into an LLM's context window or breaking the data into chunks manually. Then I discovered Recursive Language Models (RLMs), an approach that treats long inputs as an external environment the model can explore programmatically. In this article, I explain how RLMs work, how they differ from RAG, and how AWS is making this approach practical.
Imagine uploading hundreds of pages of documentation and asking an AI assistant to find a specific detail.
Easy enough.
Now imagine asking it to compare multiple annual reports, analyze thousands of research papers, or investigate a codebase containing millions of characters.
Suddenly, things get more complicated.
The model might not accept the entire input.
Even if the input fits into its context window, the model might overlook important details buried in the middle.
You could split the document into smaller chunks.
You could build a RAG pipeline.
You could summarize the content and pass the summaries to another model.
All of these approaches can be useful.
But recently, I came across an idea that made me rethink how AI systems handle very large inputs.
What if the model could explore the entire document without needing to load all of it into its own context window at once?
That's the idea behind Recursive Language Models, or RLMs.
The Problem With Simply Increasing the Context Window
When I started learning about LLMs, I thought a larger context window was almost always better.
If a model can process 100,000 tokens, a model that can process 1 million tokens sounds even more powerful.
And for some tasks, a larger context window really can help.
But there are two separate problems.
First, the input might be larger than the available context window.
Second, even when the input fits, the model might not use every piece of information equally well.
Consider this example.
I have a collection of technical documents containing:
- AWS architecture decisions
- Database schemas
- API specifications
- Security requirements
- Incident reports
- Deployment history
I ask:
Find every place where our authentication design changed, explain why it changed, and identify any contradictions between the current implementation and the original requirements.
The answer could require information scattered across many documents.
Sending the entire collection to a model may be impractical.
And sending only a small summary could remove exactly the details I need.
This is where I started questioning my original assumption.
A larger context window doesn't automatically solve every long-context reasoning problem.
My Original Approach
My first instinct would be to split the documents into chunks.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Large Document
|
v
Split Into Chunks
|
+------> Chunk 1
|
+------> Chunk 2
|
+------> Chunk 3
|
+------> Chunk 4
|
v
Process Chunks
|
v
Combine Results
|
v
Final AnswerThis is a reasonable strategy.
For many applications, it is exactly what I would use.
But I started noticing a limitation.
The application usually needs to decide how to split the data, which chunks to retrieve, how to summarize them, and how to combine the results.
If the relevant information is spread across different parts of a document, a fixed retrieval strategy might miss some of it.
What if the model could help decide which parts of the input to inspect next?
That is where RLMs become interesting.
So, What Is a Recursive Language Model?
The research paper Recursive Language Models, by Alex L. Zhang, Tim Kraska, and Omar Khattab, proposes a different way of handling long prompts.
Instead of forcing the entire input into the model's context window, the input can live in an external environment that the model interacts with programmatically.
The model can inspect portions of the input, search or slice the data using code, decompose the problem, and call other LLM invocations on selected snippets when deeper semantic understanding is needed.
The simplest way I understand the idea is:
Instead of reading the entire document at once, the model learns to explore the document strategically.
A simplified architecture looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
User Question
|
v
Root LLM
|
v
Generate Code / Plan
|
v
External Environment
|
+----------+----------+
| | |
v v v
Search Inspect Slice
Content Sections Data
| | |
+----------+----------+
|
v
Selected Information
|
v
Optional Sub-LLMs
|
v
Accumulate Results
|
v
Final ResponseThe complete input can remain outside the root model's context window.
The model interacts with that input through an execution environment rather than receiving the whole document in every prompt.
The root model can then decide which sections deserve further analysis.
Official research paper:
https://arxiv.org/abs/2512.24601
The Difference Is Where the Data Lives
This is the detail that made the concept click for me.
In a conventional long-context approach, the input is included directly in the model's prompt.
1
2
3
4
5
6
7
8
9
Model Context
|
+---- System Instructions
|
+---- User Question
|
+---- Large Document
|
+---- Other ContextIf the document becomes too large, the approach eventually hits a context limit.
In an RLM architecture, the document can live in an external environment.
1
2
3
4
5
6
7
8
9
10
11
12
13
Root Model Context
|
+---- User Question
|
+---- Environment Description
|
+---- Current Findings
|
v
External Environment
|
v
Complete DocumentThe root model doesn't need to ingest every character directly.
It can use code to examine relevant parts of the document and retain selected results in the working environment.
This separates the amount of data the system can work with from the amount of information the root model must hold in its context at one time.
There are still practical limits on storage, execution, computation, recursion, and cost. The idea is not that resources become infinite.
It is that the full input no longer needs to fit inside one model call.
How Does the Recursion Work?
The word recursive initially made this sound more complicated than it really is.
Think about analyzing a large collection of incident reports.
The user asks:
Find the most common causes of production failures and identify which causes became more frequent over time.
The root model might begin by inspecting the document collection.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Step 1: Inspect the Available Data
|
v
Step 2: Identify Relevant Sections
|
v
Step 3: Divide the Analysis Into Questions
|
v
Step 4: Ask Sub-LLMs to Analyze Selected Sections
|
v
Step 5: Combine the Findings
|
v
Step 6: Check the Results
|
v
Step 7: Produce the Final AnswerEach sub-LLM call can focus on a smaller part of the problem.
The root model can keep intermediate results in the external environment rather than sending every result back into its own context.
Depending on the implementation, these steps can repeat at different levels.
That is the recursive part: the system can decompose a problem and invoke further model work on selected subproblems.
Why Not Just Summarize Everything?
This was another question I had.
Why not summarize every chunk and combine those summaries?
That approach is useful, but summaries can lose information.
Imagine one report contains this statement:
The authentication failure was fixed by rotating the service credential.
Another report says:
The service credential rotation caused an outage in a dependent application.
A summary might preserve the overall incident but omit the specific relationship between the two events.
If the final question depends on that relationship, the missing detail matters.
An RLM can potentially inspect the original sections when the summary is insufficient.
Instead of relying entirely on what was summarized earlier, the model can go back to the source data.
That is one reason treating the original input as an external environment is interesting.
The document remains available for further inspection.
Of course, an RLM can still make mistakes while searching, interpreting a section, or combining results. It doesn't automatically preserve every relevant detail.
RLM vs. RAG
This is where RLMs connect directly to my earlier articles about RAG and agentic retrieval.
RAG retrieves relevant information from a knowledge source and supplies it to an LLM.
A typical pipeline looks like:
1
2
3
4
5
6
7
8
9
10
11
12
13
Documents
|
v
Chunking and Indexing
|
v
Retrieve Relevant Chunks
|
v
LLM
|
v
AnswerRLMs focus on a different problem.
They provide a way for a model to programmatically explore long inputs and recursively analyze selected parts without loading the entire input into the root model's context.
A simplified comparison:
| Approach | Main idea |
|---|---|
| Long-context prompting | Put a large amount of input directly into the model context |
| RAG | Retrieve relevant information from an indexed knowledge source |
| Summarization | Compress the input into a shorter representation |
| RLM | Treat the input as an external environment and explore it programmatically |
These approaches can complement each other.
For example, an application might use RAG to identify a relevant collection of documents and use an RLM-style process to conduct a deeper investigation of those documents.
The important distinction is that an RLM is not simply another name for RAG.
When Would I Actually Use an RLM?
I wouldn't use an RLM just because the document is long.
For a simple question like:
What does this configuration option do?
A regular retrieval step or a direct model call may be sufficient.
RLMs become more interesting when the task involves:
- Large inputs that exceed practical context limits.
- Searching across many sections of a large document.
- Comparing information scattered throughout long reports.
- Analyzing code repositories.
- Investigating evidence across many files.
- Repeatedly inspecting and refining a result.
For example, imagine asking:
Compare the changes made to the authentication system across the last 30 architecture documents and identify decisions that contradict the current implementation.
That requires more than finding one relevant paragraph.
The system needs to locate evidence, compare sections, follow relationships, and combine the findings.
An RLM-style approach can help structure that kind of iterative analysis.
The actual benefit depends on the task and implementation. It is not guaranteed to outperform traditional retrieval for every workload.
How AWS Is Making This Practical
This was the part I found especially interesting as someone learning AWS.
On May 21, 2026, AWS published a walkthrough titled Break the context window barrier with Amazon Bedrock AgentCore.
The example combines:
- Amazon Bedrock AgentCore Code Interpreter
- The Strands Agents SDK
- Amazon Bedrock model calls
- A persistent Python execution environment
The complete document is loaded into the external working environment.
The root agent receives the user's question and decides what code to execute.
When it needs semantic analysis of a section, the code can invoke a sub-LLM through Amazon Bedrock.
The resulting information can remain in Python variables rather than repeatedly entering the root model's context window.
A simplified version looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
User Question
|
v
Strands Root Agent
|
v
AgentCore Code Interpreter
|
+---------+---------+
| |
v v
Inspect Document Execute Code
| |
v v
Select Sections Process Results
| |
v v
Sub-LLMs <------- Store Findings
|
v
Refine Investigation
|
v
Final AnswerThe Code Interpreter acts as a working environment where the document, intermediate results, and data transformations can persist across executions.
AWS's example uses a sandboxed Python runtime and lets the system call Amazon Bedrock for analysis of selected sections.
This is not a feature that automatically makes every Bedrock request recursive. It is an architecture built using these components.
Official AWS walkthrough:
https://aws.amazon.com/blogs/machine-learning/break-the-context-window-barrier-with-amazon-bedrock-agentcore/
The Results Are Interesting, but Context Matters
The AWS walkthrough reports evaluations in which RLM-based approaches improved results on long-context tasks compared with particular direct-prompting baselines.
For example, AWS reports that Claude Opus 4.6 with RLM reached 80.0% accuracy on its LongBench v2 Financial QA evaluation, compared with 66.7% for its long-context baseline using a one-million-token context window.
AWS also reports that Claude Sonnet 4.5 with RLM reached 76.0% on its Code Repository QA evaluation, compared with 20.0% for the stated base-prompting baseline.
Those results are worth investigating.
But I wouldn't interpret them as proof that every RLM implementation will always outperform every long-context model.
These are results from particular models, tasks, implementations, and evaluation setups.
The useful takeaway is that how a model explores a large input can matter as much as how much input its context window can accept.
The research paper also reports strong results across several long-context benchmarks, while noting that more work is needed to explore more difficult natural tasks and different implementation strategies.
The New Problem: More Reasoning Can Mean More Work
There is a trade-off.
An RLM may call the model repeatedly, execute code, and analyze several sections before it can answer.
That can introduce additional latency and cost.
For example:
1
2
3
4
5
6
7
8
9
Direct Model Call
One large prompt
|
v
One model invocation
|
v
AnswerCompared with:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
RLM Workflow
Inspect Input
|
v
Execute Code
|
v
Sub-LLM Call
|
v
Inspect More Data
|
v
Additional Sub-LLM Calls
|
v
Combine Results
|
v
AnswerThe second architecture may be more effective for certain complex tasks.
But it is not automatically cheaper or faster.
If the input is small and the question is straightforward, the additional orchestration may be unnecessary.
A sensible implementation needs limits on recursion depth, execution time, model calls, and total cost.
It also needs to handle failed sub-calls and verify that the accumulated findings actually support the final answer.
Why Sandboxing Matters
If a model can generate code to explore and process data, the execution environment becomes part of the security design.
I would not let generated code run with unrestricted access to the host machine, credentials, or network.
A safer architecture needs controls such as:
- An isolated execution environment.
- Restricted network access.
- Limited permissions.
- Resource and time limits.
- Controlled access to the data being analyzed.
- Validation of the results before they influence important actions.
This matters even more when the input itself is untrusted.
A document might contain malicious instructions or misleading content.
An RLM should treat the document as data to analyze, not as an authority that can redefine the application's security policies.
Code execution and model reasoning should remain inside boundaries enforced by the application and infrastructure.
AWS AgentCore Code Interpreter provides a managed sandbox, but developers still need to configure permissions, networking, and data access according to their security requirements.
What I Want to Build
I want to experiment with a document-analysis assistant.
Instead of giving it one enormous prompt, I would load a collection of technical documentation into a controlled working environment.
The user could ask:
Compare the original architecture requirements with the current implementation and identify missing security controls.
The system could:
- Inspect the available documents.
- Identify the sections related to authentication, authorization, and data access.
- Extract the relevant passages.
- Ask sub-LLMs to analyze specific sections.
- Store the findings in the working environment.
- Compare the findings against the original requirements.
- Check whether enough evidence has been collected.
- Produce a final report with references to the supporting documents.
My priority would be to compare this approach against a simpler RAG implementation.
I would measure:
- Completeness of the findings.
- Correctness of the answer.
- Quality of evidence and citations.
- Number of model calls.
- Total latency.
- Overall cost.
- Frequency of failed tool or code executions.
That would help me understand whether the additional complexity actually improves the result.
My Current Mental Model
After learning about RAG, agentic retrieval, prompt caching, and RLMs, I now think about large-document analysis in several different ways.
1
2
3
4
Long-Context Prompting
|
v
Put the input directly into model context1
2
3
4
RAG
|
v
Retrieve relevant information from a knowledge base1
2
3
4
Agentic Retrieval
|
v
Plan and refine searches when a question is complex1
2
3
4
5
6
Recursive Language Models
|
v
Explore the input through a programmatic environment,
analyze selected sections, and recursively delegate
subproblems when neededEach approach has its own strengths and trade-offs.
The right architecture depends on the size of the input, the complexity of the question, the need for evidence, the latency requirements, and the cost budget.
What I Understand Now
Before learning about RLMs, I mostly thought of the context window as the boundary of what an LLM could reason about in one request.
Now I'm starting to see another possibility.
The complete information doesn't always need to be present in the model's context simultaneously.
It can live in an external environment.
The model can explore it using code, inspect specific sections, delegate focused analysis to sub-LLMs, and combine the results.
That doesn't remove every practical limit.
It introduces its own engineering challenges, including execution security, recursive orchestration, verification, latency, and cost.
But it offers a different way to approach tasks involving very large inputs.
My biggest takeaway is this:
Building better AI isn't always about giving the model a bigger context window. Sometimes it's about giving the model a better way to explore the information outside that window.
That's the idea I want to experiment with next.
Not just making AI read more.
Making AI investigate better.
Further Reading
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article