Prompt Compression using Amazon Bedrock :: Reduce RAG costs
With genAI applications increasingly requiring to use more bigger prompts, multimodal and involving agentic flows there is a need for techniques to optimize costs for these workloads.

With genAI applications increasingly requiring to use more bigger prompts, multimodal and involving agentic flows there is a need for techniques to optimize costs for these workloads. Prompt compression is one such technique with bigger LLMs like Claude4 Sonnet/Opus that achieves 7.5x compression and reduces the cost up to 3.5x. The technique includes using a cheaper LLM to compress the prompt for the more expensive/capable LLM.
Challenges with longer prompts :
Prompts with genAI applications are becoming increasingly complex especially with enlarged context for models, building agents or using Multi-modal RAG. Also newer models like Claude 3.7 Sonnet or Meta's Llama 3.2 Vision Instruct are multimodal, supporting much bigger context window resulting in much bigger prompts. As we scale applications with bigger FMs (foundation models) and complex prompts we can encounter following challenges :
1/ High Latency
2/ Limited context windows
3/ Forgetting contexts
4/ High Costs
5/ Slower performance
Pricing is directly proportional to your input and output token numbers.
Optimize Costs using Prompt Compression
There are various ways to optimize costs for your genAI workload, some known techniques to optimize costs on Amazon Bedrock are :
- Choosing at Infrastructure :
Could be at batched inference or Choosing provisioned throughput (Guaranteed throughput at a fixed cost) over On-demand (great for prototyping or smaller workloads) to on Amazon Bedrock can help reducing costs of your genAI workload significantly. In order to make most this , you need to have stable workload or have estimated usage in advance. You can choose Provisioned thoughput with flexible commitment 1month or 6 months
- Optimzing at FM : choosing smaller FMs ; not one model fits all, choosing smaller model over bigger ones can significantly reduce costs
- Optimzing at prompts :
At individual prompt level , you can adhere to prompt engineering guidelines. You can check best practices ; or follow this blog for best practices for Claude on Amazon Bedrock.
But on high level some best practices are :
1/ For complex requests, try a step-by-step approach vs. cramming everything into one prompt.
2/ Experiment with different wording to express ideas in fewer tokens.
3/ Avoid huge blocks of text. Summarize previous parts of a chat before moving on.
however, these best practices work for individual prompts which can become quite complex to manage really fast.
Imagine a usecase like agent assist in CX where realtime agent recommendations are generated based on live call transcriptions (check realtime agent assist blog ) ; in such usecases it's hard to follow prompt engineering best practices where call transcriptions are used as context or multi agent orchestrations where tools are chosen depending upon the user query request.
We need a technique that goes beyond optimizing for individual prompts.
In this post we will discuss one such technique that helps reduce costs when using bigger FMs and achieve similar accuracy but at lower costs.
Prompt Compression
Prompt compression is a method for efficiently condensing input prompts while preserving essential information. Its a cost optimization technique. This technique is particularly helpful in usecases like CX, doc summarisation, agents etc where large amount of tokens are given as input to larger models like Claude Opus / Sonnet etc that have high input/output token costs. With launch of powerful multimodal models like Sonnet 3.5 , agentic FMs we expect to see much more complex usecases involving larger prompts hence technique like this can become a way to optimise costs.
Prompt compression technique for LLMs that achieves 7.5x token compression and reduces the cost up to 3.5x. The technique includes using a cheaper LLM to compress the prompt for the more expensive/capable LLM.

Getting started with Prompt Compression
As we are now familiar with why we need prompt compression and how its different from prompt engineering. Let's look into how we can get started.
Amazon Bedrock allows you to create system prompts and we will be leveraging this capability to achieve prompt compression for our workloads.
Note - As Claude Sonnet 3.5 is also now available , at time of authoring we have used Claude 3 , you can use this technique for Claude 3.5. Do let us know your feedback in comments
Why need compression?
For this post we are using Claude 3 Haiku and Claude 3 Sonnet
Model | Input Costs | Output Costs |
|---|---|---|
Claude3 Haiku | $0.00025 | $0.00125 |
Claude3.7 Sonnet | $0.003 | $0.015 |
As smaller models like Haiku can be significantly cheaper and we can selectively use bigger models for specific objective to get desired results.
Using smaller model as compressor model can reduce input token size and still help in achieving similar results but at lesser costs.
Generating compressed prompt

Original Prompt (1682 words ~2897 tokens) :
text
" Please reference the following examples to answer the math question, Question: Angelo and Melanie want to plan how many hours over the nextweek they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize.They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet.If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day? Let's think step by step Angelo and Melanie think they should dedicate 3 hours to each of the 2 chapters, 3 hours x 2 chapters = 6 hours total. For the worksheets they plan to dedicate 1.5 hours for each worksheet, 1.5 hours x 4 worksheets = 6 hours total. Angelo and Melanie need to start with planning 12 hours to study, at 4 hours a day, 12 / 4 = 3 days. However, they need to include time for breaks and lunch. Every hour they want to include a 10-minute break, so 12 total hours x 10 minutes = 120 extra minutes for breaks. They also want to include 3 10-minute snack breaks, 3 x 10 minutes = 30 minutes. And they want to include 30 minutes for lunch each day, so 120 minutes for breaks + 30 minutes for snack breaks + 30 minutes for lunch = 180 minutes, or 180 / 60 minutes per hour = 3 extra hours. So Angelo and Melanie want to plan 12 hours to study + 3 hours of breaks = 15 hours total. They want to study no more than 4 hours each day, 15 hours / 4 hours each day = 3.75 They will need to plan to study 4 days to allow for all the time they need. The answer is 4 Question: Mark's basketball team scores 25 2 pointers, 8 3 pointers and 10 free throws. Their opponents score double the 2 pointers but half the 3 pointers and free throws. What's the total number of points scored by both teams added together? Let's think step by step Mark's team scores 25 2 pointers, meaning they scored 252= 50 points in 2 pointers. His team also scores 6 3 pointers, meaning they scored 83= 24 points in 3 pointers They scored 10 free throws, and free throws count as one point so they scored 101=10 points in free throws. All together his team scored 50+24+10= 84 points Mark's opponents scored double his team's number of 2 pointers, meaning they scored 502=100 points in 2 pointers. His opponents scored half his team's number of 3 pointers, meaning they scored 24/2= 12 points in 3 pointers. They also scored half Mark's team's points in free throws, meaning they scored 10/2=5 points in free throws. All together Mark's opponents scored 100+12+5=117 points The total score for the game is both team's scores added together, so it is 84+117=201 points The answer is 201 Question: Bella has two times as many marbles as frisbees. She also has 20 more frisbees than deck cards. If she buys 2/5 times more of each item, what would be the total number of the items she will have if she currently has 60 marbles? Let's think step by step When Bella buys 2/5 times more marbles, she'll have increased the number of marbles by 2/560 = 24 The total number of marbles she'll have is 60+24 = 84 If Bella currently has 60 marbles, and she has two times as many marbles as frisbees, she has 60/2 = 30 frisbees. If Bella buys 2/5 times more frisbees, she'll have 2/530 = 12 more frisbees. The total number of frisbees she'll have will increase to 30+12 = 42 Bella also has 20 more frisbees than deck cards, meaning she has 30-20 = 10 deck cards If she buys 2/5 times more deck cards, she'll have 2/5*10 = 4 more deck cards. The total number of deck cards she'll have is 10+4 = 14 Together, Bella will have a total of 14+42+84 = 140 items The answer is 140 Question: A group of 4 fruit baskets contains 9 apples, 15 oranges, and 14 bananas in the first three baskets and 2 less of each fruit in the fourth basket. How many fruits are there? Let's think step by step For the first three baskets, the number of apples and oranges in one basket is 9+15=24 In total, together with bananas, the number of fruits in one basket is 24+14=38 for the first three baskets. Since there are three baskets each having 38 fruits, there are 3*38=114 fruits in the first three baskets. The number of apples in the fourth basket is 9-2=7 There are also 15-2=13 oranges in the fourth basket The combined number of oranges and apples in the fourth basket is 13+7=20 The fourth basket also contains 14-2=12 bananas. In total, the fourth basket has 20+12=32 fruits. The four baskets together have 32+114=146 fruits. The answer is 146 Question: You can buy 4 apples or 1 watermelon for the same price. You bought 36 fruits evenly split between oranges, apples and watermelons, and the price of 1 orange is $0.50. How much does 1 apple cost if your total bill was $66? Let's think step by step If 36 fruits were evenly split between 3 types of fruits, then I bought 36/3 = 12 units of each fruit If 1 orange costs $0.50 then 12 oranges will cost $0.50 * 12 = $6 If my total bill was $66 and I spent $6 on oranges then I spent $66 - $6 = $60 on the other 2 fruit types. Assuming the price of watermelon is W, and knowing that you can buy 4 apples for the same price and that the price of one apple is A, then 1W=4A If we know we bought 12 watermelons and 12 apples for $60, then we know that $60 = 12W + 12A Knowing that 1W=4A, then we can convert the above to $60 = 12(4A) + 12A $60 = 48A + 12A $60 = 60A Then we know the price of one apple (A) is $60/60= $1 The answer is 1 Question: Susy goes to a large school with 800 students, while Sarah goes to a smaller school with only 300 students. At the start of the school year, Susy had 100 social media followers. She gained 40 new followers in the first week of the school year, half that in the second week, and half of that in the third week. Sarah only had 50 social media followers at the start of the year, but she gained 90 new followers the first week, a third of that in the second week, and a third of that in the third week. After three weeks, how many social media followers did the girl with the most total followers have? Let's think step by step After one week, Susy has 100+40 = 140 followers. In the second week, Susy gains 40/2 = 20 new followers. In the third week, Susy gains 20/2 = 10 new followers. In total, Susy finishes the three weeks with 140+20+10 = 170 total followers. After one week, Sarah has 50+90 = 140 followers. After the second week, Sarah gains 90/3 = 30 followers. After the third week, Sarah gains 30/3 = 10 followers. So, Sarah finishes the three weeks with 140+30+10 = 180 total followers. Thus, Sarah is the girl with the most total followers with a total of 180. The answer is 180 Question: Sam bought a dozen boxes, each with 30 highlighter pens inside, for $10 each box. He rearranged five of these boxes into packages of six highlighters each and sold them for $3 per package. He sold the rest of the highlighters separately at the rate of three pens for $2. How much profit did he make in total, in dollars? Let's think step by step Sam bought 12 boxes x $10 = $120 worth of highlighters. He bought 12 * 30 = 360 highlighters in total. Sam then took 5 boxes × 6 highlighters/box = 30 highlighters. He sold these boxes for 5 * $3 = $15 After selling these 5 boxes there were 360 - 30 = 330 highlighters remaining. These form 330 / 3 = 110 groups of three pens. He sold each of these groups for $2 each, so made 110 * 2 = $220 from them. In total, then, he earned $220 + $15 = $235. Since his original cost was $120, he earned $235 - $120 = $115 in profit. The answer is 115 Question: In a certain school, 2/3 of the male students like to play basketball, but only 1/5 of the female students like to play basketball. What percent of the population of the school do not like to play basketball if the ratio of the male to female students is 3:2 and there are 1000 students? Let's think step by step The students are divided into 3 + 2 = 5 parts where 3 parts are for males and 2 parts are for females. Each part represents 1000/5 = 200 students. So, there are 3 x 200 = 600 males. And there are 2 x 200 = 400 females. Hence, 600 x 2/3 = 400 males play basketball. And 400 x 1/5 = 80 females play basketball. A total of 400 + 80 = 480 students play basketball. Therefore, 1000 - 480 = 520 do not like to play basketball. The percentage of the school that do not like to play basketball is 520/1000 * 100 = 52 The answer is 52 Question: Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make? "
Compression Prompt : Using System Prompt using Amazon Bedrock
We use this sample compression prompt together with a smaller model (claude3 Haiku) to generate a compressed prompt that will be given as input to bigger model like Sonnet3
plaintext
Compress the given text to short expressions, and such that you can reconstruct it as close as possible to the original. Unlike the usual text compression, I need you to comply with the 5 conditions below:1. You can ONLY remove unimportant words.2. Do not reorder the original words.3. Do not change the original words.4. Do not use abbreviations or emojis.5. Do not add new words or symbols.Compress the origin aggressively by removing words only. Compress the origin as short as you can, while retaining as much information as possible.
Compressed Input
Here is the compressed prompt of Original prompt :: (words - 132 words ~383tokens v/s original - 1682 words ~2897tokens) — 7.56x compression
plaintext
Angelo, Melanie: 2 chapters, 4 worksheets. 3 hrs/chapter, 1.5 hrs/worksheet. 4 hrs/day, breaks, snacks, lunch. 4 days total.Mark's team: 25 2-pts, 8 3-pts, 10 FTs. Opp: 2x 2-pts, 1/2 3-pts, 1/2 FTs. 201 total points.Bella: 60 marbles, 2x marbles to frisbees, 20 more frisbees than deck cards. Buy 2/5 more. 140 items total.4 baskets: 9, 15, 14 fruits each. 4th basket 2 less. 146 fruits total.4 apples = 1 watermelon. 36 fruits, 12 each. $0.50/orange. $66 total. $1/apple.Susy: 100 followers, +40, +20, +10. Sarah: 50 followers, +90, +30, +10. Sarah 180 followers.Sam: 12 boxes, $10 each. 5 boxes, 6 pens, $3/box. Rest, 3 pens/$2. $115 profit.2/3 males, 1/5 females play basketball. 3:2 male:female. 52% don't play.Josh: $80k house, $50k repairs. Value up 150%. Profit?
Now instead of using 1682 input tokens for Sonnet , we will provide 132 words to Sonnet3.
Evaluating Compression
For this post we are using Document summarisation usecase. We compare Sonnet3 output with Original Prompt (1682 words) v/s Compressed Prompt (132 Words)
Results from Original prompt
plaintext
To solve this problem, we need to find the selling price of the house after the repairs and then calculate the profit by subtracting the total cost from the selling price.Given information:- Josh buys a house for $80,000.- He spends $50,000 on repairs.- The repairs increased the value of the house by 150%.Step 1: Find the value of the house after the repairs.Initial value of the house = $80,000Increase in value due to repairs = 150% of $80,000Increase in value due to repairs = 1.5 × $80,000 = $120,000Value of the house after repairs = Initial value + Increase in valueValue of the house after repairs = $80,000 + $120,000 = $200,000Step 2: Calculate the total cost incurred by Josh.Total cost = Cost of the house + Cost of repairsTotal cost = $80,000 + $50,000 = $130,000Step 3: Calculate the profit by subtracting the total cost from the selling price.Profit = Selling price - Total costProfit = $200,000 - $130,000 = $70,000Therefore, Josh made a profit of $70,000 by flipping the house.
Results from compressed prompt
plaintext
To calculate Josh's profit, we need the following information:- Initial cost of the house: $80,000- Cost of repairs: $50,000- Increase in value: 150%First, let's calculate the total cost incurred by Josh:Total cost = Cost of the house + Cost of repairsTotal cost = $80,000 + $50,000 = $130,000Now, we need to calculate the new value of the house after the 150% increase:New value = Initial cost × (1 + Percentage increase)New value = $80,000 × (1 + 1.5) = $80,000 × 2.5 = $200,000Finally, we can calculate Josh's profit by subtracting the total cost from the new value:Profit = New value - Total costProfit = $200,000 - $130,000 = $70,000Therefore, Josh made a profit of $70,000.
At Human evaluation results look similar. Now let's evaluate impact on costs we had using compression
Evaluating costs
Assuming a Doc Summarisation usecase with 500,000 documents to be summarised monthly
input tokens for 500,000 docs summarised monthly (original doc 1682 words/ 2897 tokens)

Conclusion
Prompt Compression offers a way to avail bigger models by using compression prompt helps achieve similar results but at lower costs using Amazon Bedrock system
Next steps :
This pattern can be further extended into emerging patterns like Contextual compression -- highly useful for emerging multi-agenting solutions. get started here
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article