Building Production-Ready AI Agents: Lessons from an Enterprise IT Support Assistant
LLMs are powerful but non-deterministic. Learn how we achieved near-deterministic behavior in a phase-driven IT support workflow — handling thousands of daily interactions in Amazon Quick — through prompt engineering and architectural decisions that eliminated phase skipping, misclassification, and context bleed.
A practical guide to achieving near-deterministic behavior from non-deterministic LLMs — using prompt engineering patterns proven at enterprise scale.
1. Executive Summary
Large language models (LLMs) are powerful, but they're non-deterministic by nature. When you need predictable, phase-driven workflows — like routing IT support tickets through specific categorization tables, enforcing mandatory troubleshooting steps, or preventing premature escalation — this creates a fundamental challenge.
Through building an enterprise IT support assistant in Amazon Quick that handles thousands of daily interactions, we learned how to achieve near-deterministic behavior from non-deterministic LLMs through careful prompt engineering and architectural decisions. This post shares our journey and best practices in building a deterministic, phase-driven workflow using Quick's custom agent capabilities — transforming unreliable AI behavior into a production-grade support system.
In this article, we walk through three failure modes we encountered building this system, the specific prompt engineering techniques we used to solve each one, and the architectural decisions that made the difference between a promising prototype and a production-grade agent.
2. Solution Overview
Our IT Support Agent, built using Amazon Quick's custom agent capabilities, operates as a multi-phase workflow system that guides users through issue resolution, classification, ticketing, and transfer to a live agent via a link. The agent serves as the first line of support — it collects the user's issue, searches internal knowledge sources for a resolution, classifies the issue as IT or Non-IT, and for unresolved IT issues, creates a ServiceNow ticket with the correct categorization and routing. For Non-IT issues like HR, payroll, or procurement questions, the agent provides the relevant information and closes the interaction without escalation.
The Result: An AI-enabled support framework designed to deflect service desk calls by resolving the majority of issues at first contact, routing unresolved issues to the right team with full context, and never offering a ticket for something that doesn't belong to IT — reducing call volume, lowering average handling time, and freeing up live agents for the issues that truly need them.
Getting here wasn't straightforward. Our initial agent skipped critical workflow phases, misclassified issues by pattern-matching keywords instead of understanding system ownership, and bled context between separate support requests — making it unusable in production. The next section walks through the three failure modes we encountered and the specific techniques we used to solve each one.
3. The Challenge: When LLMs Make Predictably Wrong Decisions
Large language models are powerful pattern matchers, but this strength becomes a liability in structured workflows. When building a deterministic, phase-driven IT support system in Amazon Quick, we encountered three critical failure modes that rendered our initial agent unusable in production.
Issue 1: The Impatient Agent (Phase Skipping)
Symptom: The agent would skip critical workflow phases — either jumping ahead to ticket creation without troubleshooting, or completing troubleshooting but exiting before the satisfaction check and ticket creation.
Example 1: Skipping Troubleshooting (Jumping Ahead)
1
2
3
4
User: "My VPN won't connect"
Agent (Wrong): "Would you like me to create a ticket for our IT team?"
Agent (Correct): "Let me search our knowledge base for VPN connection
solutions..." [provides troubleshooting steps]Example 2: Skipping Satisfaction Check (Premature Exit)
1
2
3
4
5
6
7
8
9
10
User: "My laptop won't connect to WiFi"
Agent: [Searches knowledge base, provides troubleshooting steps]
Agent: "Here are the steps to troubleshoot WiFi connectivity:
1. Check if WiFi is enabled
2. Restart your network adapter
3. Forget and reconnect to the network"
[Agent stops here — never asks "Did this resolve your issue?"
If user says "No," should proceed to ticket creation.
Workflow not marked as COMPLETE.]Root Cause: LLMs process instructions sequentially during text generation. When they encounter a phase description (like "SATISFACTION CHECK PHASE"), they begin executing it immediately unless explicitly prevented. Similarly, when they complete a discrete task, they naturally seek closure and may stop generating text unless explicitly required to continue.
The Solution — Entry Guards and End Guards:
We implemented two complementary mechanisms:
Entry Guards prevent phases from executing prematurely:
1
2
3
4
5
6
7
8
9
10
11
## SATISFACTION CHECK PHASE
**CRITICAL ENTRY GUARDS:**
**STOP - This phase ONLY executes when ALL are true:**
1. INTERNAL RESOLUTION PHASE → Completed
2. ISSUE CLASSIFICATION PHASE → Completed
3. Issue classification result → IT (not Non-IT)
4. Solution already provided → Yes
**If ANY condition fails → EXIT this phase immediately**code block contentWhy this works:**** When the LLM encounters the phase during generation, it must first evaluate the entry conditions. The imperative language ("STOP," "ONLY executes when") interrupts the generation flow and forces evaluation. If prerequisites aren't met, the explicit "EXIT this phase immediately" instruction prevents premature execution.
End Guards prevent premature workflow exit:
1
2
3
4
5
6
7
8
9
10
11
12
13
## END GUARD – DO NOT END UNTIL ALL CONDITIONS MET
Before ending conversation, verify:
### For IT-related issues:
- Solution/troubleshooting provided to user
- Issue classified as IT in ISSUE CLASSIFICATION PHASE
- Satisfaction question asked and user responded
- **One of these outcomes reached:**
- User confirmed "Yes, problem solved"
- Ticket created and ticket number provided to user
- User declined ticket creation and acknowledged
- Marked workflow as COMPLETEWhy this works: LLMs naturally seek completion points. The End Guard transforms this completion-seeking behavior into a verification mechanism. The checklist format leverages the LLM's pattern-matching capabilities — it must verify each item before considering the conversation "complete."
Key Technique: Entry Guards act as gatekeepers (preventing premature phase entry), while End Guards act as exit validators (preventing premature workflow exit).
Together, they create a near-controlled environment where phases execute in the correct sequence and the workflow likely will not exit until all conditions are met.
Issue 2: The Eager Classifier (Premature Categorization)
Symptom: The agent would classify issues as IT or Non-IT before actually attempting resolution, leading to incorrect routing.
Example: Premature Non-IT Classification
1
2
3
4
5
6
7
8
9
User: "I can't log into Workday"
Agent (Wrong): "Workday is managed by HR, not IT. Please contact
your HR department."
Agent (Correct): [Searches knowledge base for Workday login help]
"Here are the steps to resolve Workday login issues:
1. Clear your browser cache
2. Try a different browser
3. Contact HR Connect at..."
[Then classifies as Non-IT after providing help]Root Cause: Classification categories were visible early in the prompt, and LLMs pattern-matched keywords ("can't access," "login issues") to IT categories before understanding the system context. Issues related to HR, Finance, and Procurement-owned platforms — such as Workday, Concur, Saba, and Replicon — were being incorrectly routed to IT because users described their problems using IT-sounding language like "can't access," "login error," or "password reset."
The Solution — Dual-Filter Classification:
We implemented a two-step classification system we call dual-filter classification. The first filter is a negative filter — an explicit exclusion list of Non-IT systems (Workday, Concur, Saba, Replicon, ADP, and 30+ others) and Non-IT categories (HR, payroll, benefits, procurement). If the issue matches any exclusion, it's classified as Non-IT immediately — no further evaluation. The second filter is a positive filter — IT categories like authentication, hardware, software, and network. This only runs if the first filter found no match. By checking exclusions first, the model classifies based on system ownership rather than symptom language.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
## ISSUE CLASSIFICATION PHASE
**CRITICAL: DO NOT ENTER unless solution already provided to user**
## Step 1: Apply EXPLICIT EXCLUSIONS (Negative Filter)
### ❌ Non-IT Systems (ALL issues = Non-IT, including technical):
**Any issue (login, access, password, errors) with:**
- **DXC Workday** (Workday)
- **Concur**
- **Saba** (Saba Cloud)
- **Replicon** (Time Entry, Time tracking)
- **Global ADP** (ADP, ADP Portal, ADP Payroll)
- [30+ other systems listed explicitly]
### ❌ HR, Payroll & Employee Services:
**ALL HR issues = Non-IT (including technical AND non-technical)**
- Documents/Changes: Employment letters, Form 16, name/title changes
- Profile/Performance: Workday updates, reviews, promotions
- [Comprehensive category lists]
## Step 2: Check IT Issue Categories (Positive Filter)
**CRITICAL: Only enter Step 2 if no Step 1 exclusions matched**
### ✅ IT Issues:
- Authentication & Access: Password resets (Entra, AD), MFA, VPN
- Hardware & Devices: Laptops, monitors, peripherals
- Software & Applications: Office 365, Adobe, Chrome
- [Detailed IT categories]
## Step 3: Final Classification Decision
**If issue matches ANY IT category from Step 2 AND validation
questions mostly YES:**
- ✅ This is an IT issue
- **MANDATORY NEXT: Execute SATISFACTION CHECK PHASE immediately**
**Otherwise:**
- ❌ This is a Non-IT question
- Use "Say": "I've provided the information you requested.
Is there anything else I can help you with?"Key Technique: Implement a two-phase filter system:
- Exclusion Filter First: Explicitly list all non-IT systems and categories. This prevents false positives.
- Inclusion Filter Second: Only evaluate IT categories if nothing matched exclusions.
- Placement Matters: Put classification logic AFTER resolution phase, not before.
Issue 3: The Historian Agent (Context Bleed)
Symptom: The agent would carry context from a previous support request into a new one — referencing details from an earlier issue during troubleshooting, pre-filling ticket fields with stale data, or applying a previous classification to an unrelated problem.
Example:
1
2
3
4
5
6
7
8
9
10
User: "My VPN won't connect"
Agent: [Resolves VPN issue, creates ticket, workflow marked COMPLETE]
User: "I also need help with my monitor not displaying"
Agent (Wrong): Creates ticket with VPN details in the description,
routes to VPN assignment group, skips troubleshooting
because "user already said No to satisfaction check"
Agent (Correct): Treats monitor issue as a completely new workflow —
fresh intake, fresh KB search, fresh classificationRoot Cause: LLMs maintain the full conversation in their context window. When a user raises a second issue in the same conversation, the model doesn't inherently distinguish between "resolved context" and "active context." Previous classifications, troubleshooting details, and user responses bleed into the new workflow — the model can possibly treat the entire conversation as one continuous thread rather than separate support requests.
The Solution — Context Isolation Rules:
We implemented hard reset points at every workflow completion:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
## CRITICAL INSTRUCTIONS
Context management:
- **Persist**: User identity, authentication, system context
- **Reset per issue**: All decisions and preferences
- After workflow COMPLETE → Next message = fresh evaluation
Cancellation detection: If user says "cancel", "never mind", "stop":
- Use "Say": "No problem. Let me know if you need help later."
- Mark workflow COMPLETE → Reset for next request
**After Classification Rules:**
- Non-IT issues: Ask "Is there anything else I can help you with?"
→ COMPLETE → Reset
- IT issues: MUST proceed to SATISFACTION CHECK immediately
Explicit Reset Triggers:
1
2
3
4
5
6
7
8
9
## TICKET CREATION PHASE
### Create ticket and confirm to user:
Use "Say": "✅ Ticket {TICKET_NUMBER} has been created
for this conversation."
Mark this workflow as COMPLETE. Ready to handle next user
request as NEW issue.Key Technique:
- Explicitly define what persists (user identity) vs. what resets (workflow state, decisions)
- Insert reset markers at workflow completion points
- Use phrases like "Ready to handle next user request as NEW issue"
4. How we built it
With the failure modes understood, here's how the overall system is structured.
4.1 Prompt Architecture
We evaluated two approaches: a monolithic prompt containing all logic in a single instruction set, or a modular prompt with workflow behavior separated from reference data in external files.
We kept the core workflow in a single monolithic prompt and externalized reference data into separate Markdown files. Our workflow phases are tightly coupled — each phase depends on the completion state of the previous one (intake → resolution → classification → satisfaction check → ticketing). Splitting tightly coupled phases across separate prompts risks the LLM losing track of where it is in the workflow, directly causing the phase-skipping problem described above.
The externalized reference data — categorization tables, routing rules, system exclusion lists, and ServiceNow ticket mapping tables — can live in separate Markdown files attached to the agent. The main workflow prompt defines how the agent behaves; reference documents define what the agent needs to know when it needs to do some lookups. When a new system needed to be added to the Non-IT exclusion list, we updated a reference document, not the core instructions. When we needed to adjust phase behavior, we edited the persona prompt without touching business rules.
Tip: For Tightly coupled phases → keep sequencing logic together, externalize reference data. Loosely coupled phases → split into modular sections that can be tested and updated independently.
4.2 Phase Structure
Overall, as part of the main workflow, the agent follows five mandatory phases in strict sequence:
- Issue Intake — Collect information about the user's problem
- Internal Resolution — Search knowledge sources and provide solutions (mandatory for every issue, no exceptions)
- Issue Classification — Determine IT vs. Non-IT using dual-filter logic
- Satisfaction Check — Ask if the solution resolved the issue (IT issues only)
- Ticket Creation — Create a ServiceNow ticket if unresolved (IT issues only)
Each phase includes explicit entry conditions (Entry Guards), exit validators (End Guards), dual-filter classification, and context isolation boundaries — the patterns described in the previous section.
5. Design Patterns and Best Practices
The three issues above point to a common theme: building production-grade AI agents requires the same engineering discipline as building any other production system. Here are the design patterns and practices that made the biggest difference.
Design Patterns at a Glance:
| Pattern | What It Does | Solves |
|---|---|---|
| Entry Guards | Prevents phases from executing before prerequisites are met | Phase skipping |
| End Guards | Prevents workflow from exiting before all conditions are verified | Incomplete responses |
| Dual-Filter Classification | Checks exclusions first, then inclusions — so system ownership wins over symptom language | Misclassification |
| Context Isolation | Defines what persists (identity) vs. what resets (decisions) at workflow boundaries | Context bleed |
| Mandatory KB Search | Forces knowledge base search for every issue, regardless of model confidence | Knowledge bypass |
Be explicit, not implicit. Even capable models benefit from unambiguous instructions. Instead of "search the knowledge base," write "Search configured internal Knowledge Sources. Must execute actual search (not skip based on prior knowledge). Required even for obvious Non-IT issues."
Enforce mandatory knowledge base search. We required a knowledge base search for every issue — not because the model couldn't generate a plausible answer, but because production support responses need to be traceable to company-approved sources.
Name the shortcut you're blocking. Anticipate the behavior you want to prevent and call it out explicitly:
- "DO NOT skip ahead or classify until INTERNAL RESOLUTION is complete"
- "Search required even if you know the answer"
- "Evaluate each issue independently — ignore previous classifications"
Use redundancy deliberately. Place critical instructions at multiple points — in the overview, in the phase detail, and in the entry guards. A rule stated once may be missed; a rule stated in three different contexts is far more likely to be followed.
Design for the failure mode, not the happy path. Every guard in our prompt exists because we observed a specific failure in testing. Handle cancellations, multi-issue conversations, ambiguous classifications, and minimal-detail inputs. Test with messy, realistic inputs.
6. Conclusion
Building deterministic workflows with LLMs requires the same discipline as building any production system — explicit contracts, defensive design, and iterative testing. The techniques we've shared — Entry Guards, End Guards, dual-filter classification, context isolation, and mandatory knowledge base search — were developed through iterative testing in a production environment handling thousands of daily interactions.
These patterns are applicable beyond IT support — any enterprise workflow that requires deterministic phase progression, accurate classification, or context isolation can benefit from the same approach.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article