AWS Builder Center
Efficient Code Edits with Agents

Efficient Code Edits with Agents

This post shows a new technique for editing code files when using coding assistants for better token efficiency and cost.

Sr. Gen. AI/ML Specialist

The issue with agents editing documents

The cost to generate with LLMs is dominated by the output tokens used. The price of a million output tokens can be an order of magnitude larger than the input token price. At the time of this post, for example, GPT-5.6 Sol has an output to input token price ratio of 6 (output tokens are 6x the price of the input tokens). This difference presents an opportunity.
Naive implementations of code/file editing with LLMs requires the model to generate the document with any changes, or generate the changes and ask the user to implement them. Wouldn’t it be wonderful if we could only edit the pieces of a file that need to be edited?
An approach that currently exists is a search and replace tool. The user asks the model to make an edit to a file, and the model returns the substring in the file that will be replaced, and then the text that should replace it.
<<<<<<< SEARCH
def calculate_total(items):
total = 0
for item in items:
total += item.price
return total
=======
def calculate_total(items):
“”“Calculate the sum of all item prices.”“”
total = sum(item.price for item in items)
return total
>>>>>>> REPLACE
This approach has a number of limitations, we are doing a string search/replace (ambiguity between the same text), just knowing the edits made, doesn’t allow us to reverse the operation when ambiguity is present (changing ‘r‘ to ‘i‘, doesn’t allow us to then do the reverse of the operation without switching ‘i’ characters that existed previously), and chiefly, to make an edit, the model has to generate the entire sting that it’s replacing. There has to be a better way. I want a structured way to make edits to a file, and I want the model (or agent) to be able to make multiple edits at the same time. Finally, for the observability-minded folks, I want to be able to rewind any file changes as needed.

An idea for better editing

The genesis of this idea comes from how text editors make edits to a document internally. Instead of doing a string replace, let’s add some structure to our documents and perform edits against that structure (like an index or “tokenization” of our document).
Now, we have some hurdles... LLMs don’t (natively) distinguish between documents, instructions, tools, anything really (with small caveats). But if we’re going to make edits to a document, we need a way to separate the document from general context. There are a number of options for how we can add structure to a document. We can view a document as a list of characters, words, sentences, lines, pages, etc. The “best” choice here will depend on the use-case, but when we consider the popularity of coding assistants, seems like a good middle ground is a line (I.e. instead of viewing a document as unstructured text, we will convert any document into a list of lines). To implement this (since again, LLMs don’t distinguish), we will add lines directly to the document, and this is fed into the model.
import logging 0: import logging
import os 1: import os
2:
DEFAULT = Decimal("2") 3: DEFAULT = Decimal("2")
4:
class Reconciler: 5: class Reconciler:
def __init__(self): --> 6: def __init__(self):
self.tolerance = os.getenv("TOL", DEFAULT) 7: self.tolerance = os.getenv("TOL", DEFAULT)
8:
def run(self, movements, publisher): 9: def run(self, movements, publisher):
for movement in movements: 10: for movement in movements:
publisher.publish(movement) 11: publisher.publish(movement)
return len(movements) 12: return len(movements)
Additionally, we need a way to actually edit the document. If we have a number of edits we want to make we can view the problem as denoting the edits to go from document → document*. And we can do this using 3 operations [insert, replace, delete]. Since we have structured our document into lines, we just need the model to designate the line range to perform the edit, the specific operation, and the text to be replaced. This actually works pretty well as a tool we can give to our model (or agent).

Example code file structure

0: import logging
1: import os
2:
3: DEFAULT = Decimal("2")
4:
5: class Reconciler:
6: def __init__(self):
7: self.tolerance = os.getenv("TOL", DEFAULT)
8:
9: def run(self, movements, publisher):
10: for movement in movements:
11: publisher.publish(movement)
12: return len(movements)

Example code edit

{
"edits": [
{
"range": "1,2",
"lines": []
},
{
"range": "6,8",
"lines": [
" def __init__(self, tolerance: Decimal = DEFAULT):",
" self.tolerance = tolerance"
]
},
{
"range": "10,12",
"lines": [
" publisher.publish_batch(movements)"
]
}
]
}

So what does this get us?

We should see a number of benefits with this approach. If we think about the minimum amount of information we have to generate to perform edits of any document, we need to know:
  1. Where we start the edits
  1. Where we end the edits
  1. And the actual text (the edit)
Our approach (Line operations) has these components. So can we say it’s the “best?” Well not quite. Remember we decided to structure our code into lines. So if every edit needed to change things at the scale of a line, then we might be able to start talking about “best”, but if we, for example, only wanted to change a word, our current approach would still have the LLM generate an entire line (doesn’t seem optimal to me). But we do have a number of benefits here. We don’t have any ambiguous edits (since we have the line or index). We can easily reverse any operation/edit we make. And we don’t have to generate the line(s) we are replacing.
So let’s see it in action…

Let’s evaluate

I have a few evaluations that I have performed. It sometimes gets a bit tricky, since I want to evaluate both performance (speed and token efficiency) and correctness (doesn’t matter if it’s the fastest method ever if it’s wrong). I’m not going to do much with correctness in this post since I wanted to try it out on some realistic examples (but maybe another post).
So I have a number of code files (they were generated because I couldn’t find a good dataset) of about 100 lines long, and I have a number of edits that I will ask an LLM to make (20). I’ll do this for Python and Typescript (apparently the only languages that matter anymore… aside from Rust). And I will run this evaluation as passing the code file to a model as well as an edit command (e.g. “Add FX conversion so cross-currency movements compare in USD.“). I will test 3 approaches, an LLM without any edit tool, an LLM with the search and replace tool, and an LLM with the line operations tool. So across 10 Python/Typescript files and 20 edits for each, and 2 trials each, we’ll record the latency and tokens used.

Example line operations tool spec

"toolSpec": {
"name": "line_operations",
"description": """Perform multiple line-based file edits (insert, replace, delete) in a single call. Line indices are 0-based; ranges are [start, end) — inclusive of start, exclusive of end.
insert : start == end, lines non-empty. Existing lines at/after start shift down.\nreplace: start < end, lines non-empty.\ndelete : start < end, lines == [].
\n\nExample:
file_edits([\n {"range": "2,2", "lines": ["from fastapi import FastAPI", "", "print(torch.__version__)"]},\n {"range": "3,4", "lines": []},\n {"range": "5,6", "lines": ["model.load_state_dict(torch.load('/opt/ml/model/weights.pt'))"]}\n ])""",
"inputSchema": {
"json": {
"type": "object",
"properties": {
"edits": {
"type": "array",
"description": "List of edit operations to apply.",
"items": {
"type": "object",
"properties": {
"range": {
"type": "string",
"description": "Start line index (inclusive) and the End line index (exclusive). Equal to start for an insert (comma separated). All edits, using ranges from the original document"
},
"lines": {
"type": "array",
"items": { "type": "string" },
"description": "Replacement/inserted lines without trailing newlines. Empty list means delete."
}
},
"required": ["range", "lines"]
}
}
},
"required": ["edits"]
}
}
}
Note: Since our Line operations method requires the line numbers, we will keep those, but not include them for the other two methods. There is a small input token increase when adding the lines, and I want to remove that as a potential biasing point from the other approaches.

Full results

Figure shows the difference in output tokens used to make edits to the various Python/Typescript files. Overall, both search/replace and line operations tools use fewer output tokens than the naive method (without any editing tools), but the two file editing tools also show that line-operations seem to use the fewest as well as being more tightly distributed.

Figure shows a paired bar plot comparing the input/output tokens for each of the three approaches. The notable points are that the full file approach (no file editing tool) essentially outputs the entire new code file with any updates. So the tokens are roughly equal. Adding the tool spec increases the input tokens (as now we have a tool and any structure added to the file), but more importantly, the number of (more expensive) output tokens are reduced.

Figure shows the completion latency for the 3 methods. My read of this is essentially the number of output tokens also dictates the latency (even though input tokens do add a small amount). So… since the line operation approach produced fewer tokens, it also completed the call in fewer milliseconds.

Figure shows a comparison of costs for a few select LLMs. The stacked bar is both the input and output tokens to give an overall cost for 100 edit requests.


Line operations vs. search and replace

We can look at the two file edit methods specifically. In this case we can compare the difference in output tokens used. So for the same code file, and the same code change instructions. This direct comparison shows the “distribution” of tests where the line operations method produced fewer/more output tokens compared to search and replace.
Figure shows the difference in output tokens specifically between the two file editing methods. For the same file edit task the scatter plot shows the output tokens for both methods plotted together (with line operations being on the y-axis and search/replace being on the x-axis). The diagonal dotted line is the “break-even,“ where the number of output tokens were equal for both. In general, search and replace produced more output tokens. My guess is that the main difference between the two file editing methods is that the search/replace approach requires that the text being replaced is generated as well as the new text/code.

In summary, across 400 unique edit cases, line operations reduced the median output tokens by 86%, and the median latency by 71% compared with full-file generation. Compared with search and replace specifically, line operations used 50% fewer output tokens, and responded 32% faster at the median.

In closing

Well, there we have it. A neat little method for LLMs/agents editing files that gives us token efficiencies of around 71% over the naive approach, and even 50% over the search and replace editing method. I didn’t give much of the cost difference between the methods. This is mostly because it depends on the model used and it’s pricing. I have been thinking about this off and on for more than a little while. So I’m glad to finally be able to remove it from my memory (until I have some more inspiration).
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article