Prompt injection: the flaw in the AI feature you are shipping

Every organisation we have assessed in the last year is shipping an AI feature. Most describe it as low risk because "it only reads data" or "it just calls a third-party API". Both of those sentences are where the problem starts.

The flaw, in one paragraph

A language model receives one stream of text. Your instructions and the data you asked it to work with arrive in the same channel, and the model has no reliable way to tell which is which. So if the data contains something that reads like an instruction, the model may follow it. That is prompt injection. It is not a bug in a particular model or a particular vendor's product; it is a consequence of how these systems consume input.

The reason it matters more than it sounds: your model probably has tools.

Why it is not a filter problem

The intuitive fix is to detect and strip malicious instructions before they reach the model. This does not work, for a reason worth internalising rather than accepting on authority.

Detecting an injected instruction means deciding whether a piece of natural language is an instruction or content. That is the same judgement the model is already failing to make, so you are proposing to solve the problem with another instance of the problem. Every filter of this kind has a bypass, and the bypass space is the whole of natural language plus every encoding of it. Filters raise the cost of an attack, which has value, and they do not close the class.

The industry has largely converged on this. The OWASP Top 10 for LLM Applications lists prompt injection as LLM01 and frames mitigation in terms of privilege and human oversight rather than input sanitisation. That framing is the useful one.

Where the input actually comes from

Teams reason about prompt injection as though the attacker types into a chat box. The interesting cases are the ones where they do not, because those reach systems whose owners think they are out of scope:

• A document you were sent. A CV parsed by a hiring assistant, an invoice read by an accounts-payable agent, a PDF summarised for a partner review.
• A web page your agent fetched. Anything with a retrieval or browsing step is consuming attacker-influenced text by design.
• A support ticket. The customer writes the ticket body; your triage assistant reads it.
• A record in your own database, written earlier by a user through a form nobody thought of as an injection surface.
• Your own logs, if an assistant helps engineers investigate incidents and an attacker can influence a log line.

The common shape: any text a person outside your organisation influenced, reaching a model that can then do something.

The question that determines your actual risk

Not "can our model be injected" — assume yes. The question is:

If the model followed an instruction from an attacker right now, what could it do?

Work through it concretely, per tool:

• Can it read anything the current user is not entitled to see? A retrieval index built over "all internal documents" with no per-user filtering turns an injection into a data-exfiltration primitive.
• Can it write anywhere? Send an email, post to a channel, update a record, file a ticket, open a pull request?
• Can it spend or move anything — issue a refund, adjust a limit, approve a transaction?
• Can it reach the network? An agent that can fetch a URL can encode stolen context into that URL's path. This is the exfiltration channel people miss, and rendered markdown images are the classic version of it.
• Does anything it produces get executed — SQL, shell, generated code that runs in CI?

The answers give you your blast radius, and the blast radius is the risk. A model that can only produce text in a box for one authenticated user to read is in a genuinely different position from one holding a service account.

The controls that actually hold

In rough order of how much they buy you.

1. Give the agent the user's permissions, not the application's. The single highest-value control. If the agent queries data as the requesting user, an injection can only reach what that user could already reach. Most retrieval implementations we assess index everything under one service identity, which turns a per-user access control model into a suggestion.

2. Treat every tool call as an untrusted request. Authorise it at the tool boundary the way you would authorise an HTTP request from the internet. The model deciding to call refund(order_id) is not authorisation; it is a request for authorisation.

3. Require a human for anything irreversible. Money moving, external communications, permission changes, deletions. Show the user exactly what will happen and have them confirm. This is not a UX compromise — it is the control.

4. Constrain outbound network access. Allowlist the domains a tool may reach. This closes the URL-encoded exfiltration channel more effectively than anything you can do to the prompt.

5. Keep untrusted content out of the instruction position where you can. Structural separation does not solve the problem, but combined with the above it removes the easy cases.

6. Then add input and output filtering, and log everything. Useful defence in depth. Not a control to rely on alone, and not a reason to skip 1–5.

Five of those six are engineering and permissions decisions with nothing model-specific about them. That is the point: prompt injection is managed at the architecture layer, by the same people who already know how to scope a service account.

What we do not publish

We test these systems, including for data exfiltration paths, and we do not publish working injection payloads. Explaining the class of flaw and the defences helps the people building; a copy-pasteable bypass for a specific product mostly helps the people attacking. Findings from an engagement go to your team with reproduction steps, which is where they belong.

A short checklist for your next AI review

• Does the agent query as the user, or as the application?
• List every tool. For each: read, write, spend, or network?
• What is the confirmation step for the irreversible ones?
• Where can text from outside your organisation enter the context?
• If the model exfiltrated its whole context right now, what would be in it?
• Who reviewed the system prompt for secrets? They are in there more often than you would expect.

If you cannot answer the second question from memory, that is the finding.

Next step

We threat model AI systems against the OWASP LLM Top 10 and test them adversarially, including the agent permission review that most of this article is really about. See what AI Security covers — and if the feature sits inside an existing product, it usually wants pairing with Application Security, since the interesting exfiltration paths tend to run through the app rather than the model.