AI is moving quickly from a tool that helps people do their work to technology that can potentially do some of the work itself. That distinction matters.
A generative AI tool that drafts an email or summarises a document still leaves a person in the middle of the process.
An AI agent can potentially go further. Depending on how it is designed and configured, it may be able to access systems, make decisions, initiate actions and interact with other tools.
That creates opportunities for businesses.
It also creates a different kind of risk.
The question is no longer simply whether the AI produces an accurate answer.
It becomes:
What is the AI allowed to do, what can it access and what happens when it gets something wrong?
From AI output to AI action
The first generation of workplace AI risk conversations focused heavily on information.
What data are employees putting into AI tools?
Is confidential information being exposed?
Are people using unapproved platforms?
Could AI-generated content contain errors?
Those questions remain important.
But agentic AI introduces another layer.
The system may no longer simply produce information for a human to act on.
It may be able to act itself.
That could mean sending a communication, changing information in another system, initiating a transaction, accessing data or carrying out part of a business process.
The potential risk therefore changes.
An inaccurate answer that is spotted during human review may cause relatively little damage.
An inaccurate decision made automatically by a system with access to important business processes could be much more significant.
The three questions businesses need to separate
When considering AI risk, it is useful to distinguish between three different things:
What should the AI do?
Who is responsible for it?
What actually prevents it doing something it shouldn’t?
These correspond broadly to policy, governance and control.
Policy: what should happen?
An AI policy establishes the boundaries.
It might say:
– only approved AI tools may be used
– confidential information must not be entered into certain systems
– AI-generated outputs must be checked
– certain decisions require human involvement
– employees must report AI-related incidents
This is important.
But a policy is not proof that the behaviour it describes is happening.
Governance: who is responsible?
Someone needs to own the decisions around AI.
That doesn’t necessarily mean appointing an “AI risk manager”.
It means being clear about responsibility.
Who approves AI tools?
Who decides what data they can access?
Who determines which uses are acceptable?
Who reviews changes?
Who monitors incidents?
Who reports significant risks to senior management or the board?
Without clear ownership, AI governance can quickly become everyone’s responsibility and nobody’s responsibility.
Control: what actually happens?
This is where things become more interesting.
Suppose the business says that an AI agent must not initiate certain transactions without human approval.
What makes that rule real?
Perhaps the system requires an approval before the action can proceed.
Perhaps permissions restrict what the agent can access.
Perhaps certain actions are automatically blocked.
Perhaps activity is logged and monitored.
Perhaps there is an alert when the agent behaves outside expected parameters.
These are controls because they affect what can actually happen.
Permissions become a risk issue
One of the biggest changes with AI agents is the importance of permissions.
Every system an agent can access creates another potential point of failure.
So a sensible risk assessment needs to look beyond the AI tool itself.
Ask:
– What systems can it access?
– What information can it see?
– What actions can it take?
– What decisions can it make?
– What permissions does it inherit?
– Can those permissions be restricted?
– Can they be withdrawn quickly?
– Are actions logged?
– Is there a human approval point?
– Can the agent be stopped?
This is similar to the way you would think about access controls for people.
We don’t normally give every employee unrestricted access to every business system.
The same principle should apply when the “user” is an AI system.
What happens when the AI does something unexpected?
Traditional risk assessments often focus on foreseeable failure.
But AI systems can behave in ways that are difficult to predict from a simple list of rules.
That makes testing particularly important.
For example, imagine an AI agent responsible for part of a customer-service process.
What happens when:
– the customer request is ambiguous?
– the underlying information is wrong?
– a connected system is unavailable?
– the agent receives conflicting instructions?
– someone attempts to manipulate its instructions?
– it reaches the edge of its authorised activity?
– the action it wants to take has a financial consequence?
A business should not wait for the first real incident to discover what happens.
Testing allows you to explore those situations in a controlled environment.
Human oversight needs to be meaningful
“Human in the loop” sounds reassuring.
But what does it actually mean?
If a person receives an AI recommendation and clicks “approve” without having the time, information or expertise to challenge it, the human may technically be involved without providing meaningful oversight.
Good oversight requires people to understand:
– what the system is doing
– what could go wrong
– when they need to intervene
– what evidence they need to review
– what authority they have to stop the process
The more significant the potential consequence, the more important meaningful oversight becomes.
AI risk is also operational resilience
There is another question businesses should consider.
What happens if the AI service isn’t available?
If an AI agent becomes embedded in an important business process, the business may develop a dependency on it.
That creates a resilience issue.
Imagine a process that used to require four people and now relies on an AI system to perform the first stage.
The business becomes more efficient.
But what happens if the system is unavailable?
Can the process continue manually?
Does anyone still know how to do it?
How long can the business operate without the service?
What information would be unavailable?
Are there alternative providers?
This is why technology risk cannot always be separated neatly from operational resilience.
A technology dependency can become a business dependency.
Don’t forget the wider technology chain
The same principle applies to the infrastructure supporting AI.
The service you use may depend on cloud providers, data centres, telecommunications, electricity and other infrastructure.
Your direct supplier may be reliable.
That doesn’t mean every dependency underneath it is equally resilient.
The same applies to other critical suppliers.
A business can have good relationships with its direct suppliers while remaining exposed to dependencies further down the chain.
This is why supplier risk needs to move beyond simply asking whether the supplier itself is financially stable or has a continuity plan.
The more useful question is What does our business depend on, and where are the points where those dependencies could fail?
A practical AI risk assessment
You don’t need a complicated AI risk framework to start.
Ask these questions.
1. What are we using AI for? Create a clear picture of how AI is actually being used, rather than relying solely on formal approvals.
2. What information does it access? Identify confidential, personal, commercially sensitive and otherwise important information.
3. What can it do? Map the actions and decisions the AI can make.
4. What permissions does it have? Check whether those permissions are appropriate for the role it performs.
5. What controls are in place? Look at policies, access controls, approvals, monitoring, training and incident response.
6. Have those controls been tested? Don’t assume that because the control exists, it works.
7. Who owns the risk? Make responsibility explicit.
8. What happens if it fails? Consider both individual incidents and loss of the service itself.
9. Can the business continue without it? If the answer is no, you have identified an operational dependency that needs attention.
AI doesn’t need a completely separate risk universe
This is perhaps the most important point.
AI is introducing new risks and new ways of working.
But the underlying principles of good risk management haven’t changed.
You still need to understand what matters to the business.
You still need to identify what could prevent you achieving your objectives.
You still need to understand your controls.
You still need to prioritise.
And you still need to embed what you’ve learned into the way the business operates.
That is the approach behind the TRAPS framework in my book, Strategic Risk: The TRAPS Framework for Better Business Decisions.
The framework starts with Targets, then moves through Risks, Analyse, Prioritise and Systemise.
It was designed to help businesses manage uncertainty without turning risk management into a full-time job.
AI is simply giving us another reason to apply those principles.
The question isn’t whether to use AI
The more useful questions are:
“Where are we using it?”
“What have we allowed it to do?”
“What could go wrong?”
“What controls do we have?”
“How do we know those controls work?”
And perhaps most importantly:
“If the AI makes a decision or takes an action that we didn’t expect, who is responsible?”
Those are governance questions. They’re also risk management questions.
And the sooner businesses start asking them, the easier it will be to make the most of AI without handing over more control than they intended.
Want to explore a practical approach to risk?
My book, Strategic Risk: The TRAPS Framework for Better Business Decisions, looks at how businesses can understand their critical objectives, identify risks, assess controls, prioritise action and build resilience into the way they operate.
Recent Comments