Designing AI With Limits

Artificial intelligence has spent the past few years moving from something that generates information to something that can increasingly act on our behalf.
An AI assistant can now search the web, read an email, access a calendar, work with files, interact with software, write and execute code, and in some cases make decisions or take actions across multiple systems. For users, this can feel like a natural evolution of the tools they already use. For the people building those products, however, it represents a significant change in the security model.
The distinction is important. A chatbot that gives you a wrong answer creates a reliability problem. An AI agent with access to your files, accounts, or applications can turn a wrong answer into an action.
Recent security research and incidents have made that distinction increasingly difficult to ignore. In May, the National Institute of Standards and Technology published an analysis of industry feedback on AI-agent security, noting broad agreement that agents introduce new security threats and that traditional cybersecurity practices will need to be adapted to address them.
The concern is not simply that AI models can be manipulated. It is that increasingly autonomous systems can encounter something malicious and then have the authority to do something with it.

The Problem With Trusting What an Agent Reads
One of the more significant risks is known as indirect prompt injection.
Consider an agent that has been asked to research a company. It visits a website, reads the content, and uses that information to complete the task. Hidden somewhere in that content could be an instruction designed specifically for the agent rather than the person reading the page.
A conventional browser would simply display the text. An agent may interpret it as an instruction.
NIST research into AI-agent security has demonstrated this broader class of “agent hijacking,” in which malicious instructions are embedded in websites, emails, code repositories, or other external data that an agent is expected to process. An attacker can potentially use those instructions to redirect the agent, expose sensitive information, or cause it to take unintended actions.
This creates a new problem for product designers and engineers: not everything an agent encounters should be treated as trustworthy simply because the agent was asked to read it.
That may sound obvious, but it is fundamentally different from the way many traditional software integrations work. A database query returns data. An API returns a response. A webpage contains information. An AI agent has to interpret all of those things, and its interpretation can influence what it does next.
The boundary between data and instructions therefore becomes much less clear.
When Capability Becomes Risk
The second problem is what security researchers describe as excessive agency.
An agent may be perfectly capable of performing a task while still having far more authority than it needs to perform that task safely.
An assistant that summarizes a user’s inbox needs access to email. It probably does not need permission to delete messages. An agent that prepares a purchase might need access to product information and a shopping cart, but that does not necessarily mean it should be able to complete a transaction without confirmation.
The more systems an agent can access, and the more actions it can perform within those systems, the greater the potential impact when something goes wrong.
OWASP identifies excessive functionality, excessive permissions, and excessive autonomy as common contributors to this problem. Its newer agentic-AI guidance also addresses situations where an agent can perform high-impact actions across connected systems without appropriate transactional safeguards.
This is where the conversation around AI security becomes much larger than the model itself.
The model may be only one component of the system. The surrounding tools, credentials, integrations, permissions, data sources, and interfaces can determine how much damage a compromised or misdirected agent is actually capable of causing.
What This Means for End Users
For most people, the technical details of prompt injection and agent authorization are not particularly useful. Users should not need to understand an agent’s internal architecture in order to use a product safely.
They do, however, need to understand what they are giving that product permission to do.
That means products should make important boundaries visible.
If an AI assistant can read your documents, that should be clear. If it can send an email, make a purchase, modify a file, or access another application, users should understand that before the action occurs. When an action has meaningful consequences, the interface should provide an appropriate opportunity for a person to review and authorize it.
This is more than a security feature. It is an experience-design decision.
The interface becomes part of the system of trust.
A user should not have to wonder whether an agent is currently planning something, researching something, or actually doing something. Nor should an innocuous request quietly grant an AI system broad access to information and tools that have little to do with the task at hand.
Good agentic products will make these boundaries understandable without forcing users to become security experts.

Designing for Containment
For teams building products that incorporate AI agents, the goal should not be to eliminate every possibility of failure. Complex systems will fail. Models will misunderstand instructions. External information will sometimes be malicious. Integrations will behave unexpectedly.
The more useful question is what happens when they do.
An agent should have the minimum permissions necessary to accomplish its task. Credentials should be appropriately scoped. High-impact actions should have additional safeguards. External content should be treated as potentially untrusted rather than automatically authoritative. Tool calls and other meaningful actions should be observable and auditable.
Systems should also be designed so that a mistake can be contained rather than propagated across every connected service.
These principles are increasingly reflected in emerging guidance around agent identity, authorization, and security. NIST is actively developing approaches to identifying and authorizing software agents, including controls intended to address prompt injection and the risks created when agents are given access to diverse datasets, tools, and applications.
The underlying principle is relatively simple:
An AI agent should have no more authority than the task requires.
That sounds less futuristic than many conversations about autonomous AI. It is also probably more important.
Security Is Becoming Part of the User Experience
The arrival of agentic AI does not mean that organizations should stop using these systems. It does mean that we need to rethink what responsible product design looks like when software can act with increasing independence.
Security can no longer sit entirely behind the interface.
When an AI system can access personal information, make decisions, or interact with other services, the product itself needs to communicate those boundaries. Users need meaningful control, and teams need systems that can constrain, monitor, and interrupt automated actions when necessary.
The most trustworthy AI products may ultimately be the ones that demonstrate the most restraint.
The goal is not to build agents that can do everything. It is to build agents that can do useful things within clearly defined boundaries.
As AI moves from answering questions to taking action, that distinction may become one of the defining principles of responsible digital product design.
The strongest digital experiences aren’t built on assumptions. They begin with careful observation, thoughtful questions, and a shared understanding of what matters most. If you’re ready to explore what’s possible, we’d welcome the opportunity to talk. Work with us.
Comments are closed.