AWS Builder Center
Optimizing Generative AI on AWS

Optimizing Generative AI on AWS

How to make an AI application respond faster, cost less, and handle more users

Think of it as a busy kitchen
The easiest way I found to understand AI performance is to picture a restaurant during the dinner rush. The AI model is the chef, the user’s question is the order, background information is the tray of ingredients, and network connections are the paths used to carry everything around. A bigger kitchen may help, but it will not fix a confusing order, unnecessary ingredients, or a new door being opened for every trip. The first step is to remove wasted work.
What these optimizations actually improve
Lower latency - Latency simply means how long the user waits. Caching lets the system reuse work it has already done, while token efficiency removes unnecessary words and background information. Both can help the answer appear sooner.
Higher throughput - Throughput means how many requests the system can handle in a period. A smaller model can finish easy tasks quickly, while connection pooling reuses already-open network connections. This helps the application serve more users at the same time.
Safer traffic handling - Cross-Region inference lets Amazon Bedrock use computing capacity in other approved AWS Regions when the local Region is busy. If a temporary error still happens, exponential backoff waits longer between each retry instead of repeatedly hitting an overloaded service.
Smarter retrieval - RAG lets the application search trusted documents before asking the model to answer. Good RAG tuning returns only the most useful pieces of information, so the model gets better facts without reading a pile of unrelated text.
One user request, from prompt to response
Imagine a support engineer asks, “Why is this S3 request returning AccessDenied?” Here is what may happen behind the screen:
  1. Check what can be reused - The application checks whether it has a safe saved answer. Bedrock prompt caching can also reuse repeated instructions or background text. If nothing useful is available, the request moves on.
  1. Choose the right AI model - A simple question goes to a smaller, faster model. A difficult IAM policy question may go to a more powerful model. Tests with real questions should decide which model is good enough.
  1. Find useful facts - The RAG search looks through runbooks and IAM documents. It keeps the most relevant passages and drops results that are unlikely to help.
  1. Shorten the package - Tokens are the small pieces of text an AI model reads and writes. The application removes repeated instructions, old conversation details, and weak search results so the model receives a smaller, clearer package.
  1. Reuse an open connection - Instead of building a new network connection for every request, connection pooling reuses one that is already open—similar to keeping a delivery lane ready instead of rebuilding the road each time.
  1. Look for available AWS capacity - The request uses a Bedrock inference profile. If the starting Region is busy, Bedrock may process it in another approved Region listed in that profile. Security and data-location rules still apply.
  1. Retry without making things worse - If a temporary error occurs, exponential backoff may wait about 1 second, then 2, then 4 before trying again, usually with a little randomness. Retries stop after a set limit, and permanent errors are not retried.
  1. Check whether it worked - The team measures latency, how long the user waited and throughput how many requests the system completed. It also checks answer quality and cost. A faster system is not an improvement if the answers become worse.
The practical tuning order
  • Measure where the time and money are going.
  • Remove unnecessary text, choose the right model, and improve the document search.
  • Reuse safe work and network connections, then add controlled retries.
  • Add more AWS capacity only when testing shows that capacity is the real problem.
The takeaway: Optimization is not about making every prompt as short or cheap as possible. It is about giving users a useful answer faster, serving more people reliably, and keeping the quality high.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article