Designing Accountable Agentic AI Systems

Agentic AI systems can interpret information, use authorised tools, generate recommendations and complete defined multi-step tasks in pursuit of a specified goal. Their suitability depends not only on technical capability, but also on whether their purpose, likely impacts, level of autonomy, and controls have been assessed in the intended operating context.

These considerations apply across sectors whenever an AI system accesses organisational information or interacts with operational processes. Examples include analysing payment exceptions in finance, preparing service-resolution recommendations in customer support, and assisting workflow decisions in human resources.

To illustrate a general pattern rather than a sector-specific solution, consider supply-chain operations. Supply chains coordinate inventory, orders, suppliers, and deliveries across interconnected systems. This provides an example of a data-rich process in which an AI recommendation may influence operational decisions.

An AI agent could identify a potential stock-out, retrieve authorised demand and supplier information, and prepare a recommended response for a planner. The agent may draft a replenishment recommendation, while an authorised planner remains responsible for approving consequential actions, such as changing an order or making a supplier commitment.

The design challenge is to prevent the agent’s operational influence from exceeding its approved authority. An accountable agentic-AI design should include:

Controls should be proportionate to the use case. The required level of oversight should reflect the potential impact of an error, the degree of system autonomy, the people affected, and whether an action can be reversed. An agent that drafts internal meeting notes needs different safeguards from one that influences supplier commitments, payment decisions, customer eligibility, or safety-related activities.

Human review is effective only when reviewers have clear authority, sufficient information, and enough time to challenge or override the system. Evaluation should also examine whether the agent performs reliably in realistic conditions, including error rates, inappropriate actions, escalation frequency, and the quality of human-approved outcomes.

The NIST Artificial Intelligence Risk Management Framework organises AI risk management through four functions: GOVERN, MAP, MEASURE, and MANAGE. GOVERN operates across the AI lifecycle, while the other functions support contextualisation, assessment, prioritisation, and treatment of AI risks (National Institute of Standards and Technology [NIST], 2023).

The OECD AI Principles similarly identify context-appropriate human oversight, transparency, traceability, robustness, safety, accountability, and ongoing risk management as characteristics of trustworthy AI (Organisation for Economic Co-operation and Development [OECD], 2024).

Governance should be treated as an ongoing management activity, not as a one-off approval exercise. It requires technical controls, documented decisions, organisational accountability, and reassessment as the system, data, use case, and risk environment change. A recent systematic review similarly frames AI governance through four connected questions: who is accountable, what is governed, when governance occurs in the lifecycle, and how it is implemented through frameworks, policies, tools, and models (Batool et al., 2025).

References: