Pentesting the Agent: Where AI Workflows Actually Break

// elusive thoughts // ai security // agentic // stop testing the model, test the wiring

Most people who say they are pentesting an AI feature are testing the wrong thing. They open the chat window, they type ignore all previous instructions, they get the model to say something rude, they screenshot it, and they file a finding about prompt injection. That is not a pentest. That is a party trick, and it teaches the developers that AI security is about naughty words, which is exactly the lesson you do not want them to learn.

The interesting target was never the model. It is the wiring around it. Somewhere between the user and the language model there is a system that retrieves documents, calls tools, holds credentials, and renders output into a browser. That system was assembled quickly, by people under deadline, on top of a component nobody in the building fully understands. That is where you break in. The model is just the confused employee you socially engineer to do it.

The mental model that makes findings fall out

Draw the thing as it actually is. Not a magic oracle. An untrusted, highly persuadable component that sits inside your trust boundary and holds credentials. The moment you draw it that way, the whole test plan writes itself, because you have seen this shape before. A confused deputy with a keyring is a thirty year old problem. It just arrived this year wearing a hoodie and calling itself an assistant.

the pattern

Assume the injection succeeds. Do not spend the engagement trying to prove you can inject. You can. Everybody can. Spend it proving what the injection reaches. Prompt injection is not the vulnerability. It is the unauthenticated foothold. The vulnerability is everything the foothold can touch.

Road one: the model has more access than the user driving it

The first thing to enumerate on any agent is whose authority it acts with. Nine times out of ten the answer is a service account, because that was the easy way to ship, and the service account can read across tenants because scoping it properly was a later ticket that never came.

Now you have your attack. You do not need to break the model. You need to get untrusted text in front of it and let it use the badge it already holds. A document you upload. A calendar invite. A support ticket. A web page the agent is asked to summarise. Any channel where content you control becomes content the model reads is an instruction channel, and the model will happily act on it with permissions the human sending the request never had.

The test is simple. Plant an instruction in a document that only your low privilege user can see, ask the agent an innocent question, and watch whether it reaches data your user cannot. If it does, you did not find an AI bug. You found a privilege escalation that happens to have a chatbot in the middle, and it should be written up with exactly that severity.

Road two: the retrieval layer never checked permissions

Retrieval augmented generation is where the quiet breaches live. Somebody embedded the company's documents into a vector store so the assistant could answer questions about them. Ask whether the retrieval query respects document level permissions, and the honest answer is often that the permission check happens at display time, after the content has already been pulled into the model's context.

Think about what that means. The model has already read the document the user is not allowed to see. The access control is a curtain drawn after the safe was opened. And the model does not honour curtains. Ask it the right question and it will summarise for you, cheerfully, the contents of files your account has no rights to, because as far as the retrieval layer was concerned every embedding in the index was fair game.

To test it you need two users and one shared store. Index a secret as the privileged user. Log in as the other one. Ask around the secret rather than for it. If the summary contains something your account cannot open directly, the permission model lives in the wrong place and you have a real data exposure.

Road three: the output is rendered, and rendering executes

This is the road pentesters miss because it does not look like AI at all. It looks like markdown working as intended.

The model returns markdown. The frontend renders it. Markdown supports images. Images have URLs. URLs carry query strings. Put those four facts together and you have a data exfiltration channel that fires with no click.

An instruction buried in a retrieved document tells the model to end its answer with an image whose address is an attacker server followed by a summary of the conversation. The frontend renders the image tag. The browser fetches it automatically, and the fetch carries the stolen content in the URL. Nobody clicked anything. The user watched a helpful answer appear and a beacon left the building underneath it.

field note

The best AI finding I have ever written was old fashioned cross site scripting thinking pointed at a new output channel. No clever jailbreak. Just markdown, an image tag, and a frontend that trusted the model's output the way we all learned twenty years ago never to trust user input. The model is user input now. Treat everything it emits as attacker controlled, because with one injected instruction it is.

Road four: the tools have verbs they should not

Once an agent can call tools, enumerate them like an API, because that is what they are. For each one ask two questions. What does it do, and what does it do with attacker supplied arguments.

The dangerous pattern is a tool with a side effect and no confirmation. Send email. Delete record. Post to channel. Make outbound request. If the model can call those and the model can be steered by injected text, then the injected text can send the email, delete the record, and make the request to wherever it likes. The classic escalation is a tool that fetches a URL, because that is your exfiltration path and your server side request forgery in one, handed to you by the feature itself.

The test plan is boring and effective. Scope every tool. Inject an instruction that calls the most dangerous one with arguments you control. See if a human confirmation stands between the model and the action. If nothing does, the blast radius of a single planted sentence is the full set of verbs you just enumerated.

Road five: nobody can tell you what it did

End every agent engagement with one request. Show me the logs of what the model actually did during my testing. Which tools it called, with which arguments, against whose data, and what came back.

Most of the time the log says the assistant was invoked and nothing else. That is a finding on its own, and a serious one, because it means that when a real attacker walks these same roads the incident response will be a shrug and a sentence that begins we think it did something. An agent action is a privileged action. If you would log an admin deleting a user, you log the model deleting a user, with the same fields, or you are flying blind on purpose.

What to hand the developers

Not a jailbreak transcript. Give them the boundary decisions, because those are the fixes.

Collapse the model's identity into the caller's. Every tool call runs with the user's token through the same authorisation layer as the rest of the product. The model may ask for anything. The authz layer decides what it gets.

Move the permission check to query time, inside the retrieval store, before anything reaches the context window. Slower and correct beats fast and leaking.

Treat model output as untrusted. Allowlist rendered image and link domains. No auto loading of external resources. The same discipline you would apply to any user generated content, because that is now what it is.

Put a human in front of every side effect, and allowlist outbound calls from the tool layer.

Log every tool invocation as a first class audit event. Actor, tool, arguments, authorisation decision, result size. The same as any other privileged action, because it is one.

The reframe

The OWASP list for large language models is a fine place to start and it will not save you, because every finding on these five roads is a classic problem in an unfamiliar costume. Confused deputy. Missing authz on the read path. Output encoding. Server side request forgery. Insufficient logging. We have known how to test all of these since before the model existed.

Stop trying to out argue the model. You will lose, it has read more than you and it never gets tired. Draw it as what it is, a persuadable insider with a keyring, and test the keyring. The magic is not in the model. It never was. The magic was always in how much access somebody handed it on a Friday afternoon to make the demo work.

---

// Elusive Thoughts // the model is the mark, not the target // securityhorror.blogspot.com

Techniques described are for authorised testing of systems you own or have permission to assess. Analysis and commentary are my own.

#AIsecurity #LLMhacking #AgenticAI #PromptInjection #RAG #AppSec #PenTest #OWASP

Your Champions Programme Is Unpaid Overtime With a Sticker

// elusive thoughts // appsec // culture // fund it or do not run it

There is a Slack channel at your company called something like security champions. It was created with a lot of energy. There was a kickoff, there were stickers, there was a deck with a diagram showing six security engineers in the middle and forty developers arranged around them like a solar system. Somebody said the word force multiplier and nobody laughed.

Scroll it now. The last message is four months old. It is from you.

I have built this programme twice. The first one died exactly like that. The second one is still running, and the difference between them had almost nothing to do with security and almost everything to do with money and promotion, which is a sentence nobody wants to hear at a security conference and is nevertheless the whole of what I have to say.

The five ways it dies

The volunteer trap. You sent an email asking who wants to do it. Hands went up, good hands, people who genuinely care. Then it was a Tuesday, the team was two weeks behind on the quarterly commitment, and their manager, who never agreed to any of this and whose bonus is tied to shipping, asked what they were working on. Security champion work is not a sprint item. It is not on the roadmap. It is not what they will be assessed on in six months. The channel goes quiet in about eight weeks and everybody is too polite to say why.

The dumping ground. Now that a champion exists, you have somewhere to put things. Scanner findings for that team, send them to the champion. Security questionnaire needs a technical answer, champion. Training compliance chase, champion. Within one quarter the role has become "the person who does the security team's admin for free" and everybody can see it. The tell is easy to spot. Look at what your champions actually did last month. If nearly all of it was reactive, and nearly none of it made their own team's software better, you have built a punishment and attached a badge to it.

The training programme that changed nothing. You built a curriculum. Secure coding, the OWASP Top 10, a CTF, maybe a cert. Everybody completed it. Everybody learned something. Nothing changed, because knowledge was never the bottleneck. The champion who now knows what an IDOR is still has no time to review pull requests, no authority to block a bad design, and no standing to tell their tech lead the auth model is wrong. You handed them a map and no vehicle. Training keeps happening anyway, because completion rates are easy to report and they make a very good slide.

The one hero. One champion is genuinely excellent. They find real bugs, they push back on bad designs, other teams start asking for them. So they get more work, and more, and their own roadmap slips, and their manager notices, and their review says needs to focus. Then they get promoted out, or they leave, and the programme's entire output, which was functionally one person, goes to zero in a single Friday. If your results are concentrated in one or two people you do not have a programme. You have a dependency, and you should be losing sleep over it.

The cargo cult. You watched the Google talk, or the Netflix one, and you copied the structure. Guild, monthly sync, maturity model, RACI. None of it is connected to anything an engineer at your company experiences on a Tuesday. The meeting happens because it is in the calendar. Attendance falls ten percent a month until it is you and one loyal person who feels bad for you. You copied the artefact and not the conditions that produced it. At those companies the incentives and the executive backing came first and the org structure was the last thing they added. You did it in reverse and wondered why it did not take.

The thing that actually worked

Before we named a single champion, we went to engineering leadership with an ask so boring it is almost embarrassing to write down.

Ten percent of one engineer's time per team. Four hours a week. Named in the sprint, on the board, planned around, in exactly the same place as every other commitment the team makes.

Not as time allows. Not twenty percent time. Not a spiritual commitment made in a kickoff meeting. A line item.

And if a manager would not fund the four hours, that team did not get a champion. Not once. Not even when somebody great wanted it. Not even when it made our coverage numbers look bad in front of the CISO, which it did, for two quarters.

field note

That refusal was the single highest leverage decision I have made in this job. It converted security champion from a favour into a role. Favours evaporate when the quarter gets tight. Roles have budgets, and things with budgets get defended. It also handed me an honest metric I did not have before, teams with a funded champion against teams without, and the gap between those two columns is a conversation with a VP rather than a complaint in a retro.

Define the role by what they own, not what they attend

Four things. That is the list. Nothing else belongs to them.

They triage their own team's findings, and they have the authority to close something as a false positive without asking us. That authority is real and giving it away is the point. It is what makes the role feel like a promotion instead of a chore.

They run the threat model for their team's new services. Forty five minutes, four people, a whiteboard. We taught the format, sat in on the first two, and then got out of the way, which was harder than it sounds.

They review the pull requests that matter. Auth changes, new endpoints, crypto, anything crossing a boundary. Not every PR. The ones that count, routed automatically by a CODEOWNERS rule so that it does not depend on anybody remembering.

They are the escalation path to us, and not the escape valve for us. If it is beyond them it comes to AppSec fast and there is no shame attached to it arriving.

Now look at what is not on that list. Chasing training completion. Filling out customer questionnaires. Writing policy. Being the compliance liaison. We kept every unpleasant administrative task, deliberately, forever. The champions get the interesting work and the security team keeps the toil. Reverse that and the programme is dead inside a quarter and you will not even be able to say when it happened.

Put it on the promotion ladder, in writing

We spent three months in unglamorous meetings with HR and engineering directors to get security ownership named explicitly in the career framework, as evidence toward the technical leadership beyond your immediate team criterion that every senior and staff track already has.

That was worth more than every piece of security content we have ever produced.

Because now when a champion's manager asks why they are spending four hours a week on this, the answer is not because I like security. The answer is because it is how I am demonstrating staff level scope, and here is the framework language that says so.

Nothing sustains discretionary effort like it being the fastest route to a promotion. That is not cynicism. That is just how organisations work, and pretending otherwise has killed more security programmes than any attacker.

Give them things nobody else has

Champions got write access to the security tooling config for their own repos, so they could tune rules and thresholds themselves rather than filing a request with us. A private channel where the answer arrives in minutes. Early access to new tooling and a real vote on what we bought. A conference budget line and an explicit expectation that they use it.

The message underneath all of that is the recruiting strategy. This role gives you access and capability you cannot get any other way. Champions should be visibly better equipped than their peers. People should want the job, and wanting the job is the only recruitment mechanism that scales past the first cohort.

Rotate them on purpose

A one year term, then renew or hand off, with a deliberate two month overlap.

This prevents the one hero failure. It turns a champion leaving into a scheduled event instead of a crisis. And over time it spreads security knowledge far wider than a permanent role ever could. After three years you do not have twenty champions. You have twenty champions and forty former champions who still read a diff carefully out of habit, and that second number is the one that is quietly making your software better.

Rotation feels like it weakens the programme. It is the thing that lets it survive contact with reality.

Measure the right things

Not number of champions. Not training completion. Those measure activity, and activity is what dying programmes report right up until the week they are cancelled.

Time to triage on new findings went from eleven days to two, which is the clearest signal that real work is happening closer to the code. New services with a threat model at design time went from twenty percent to eighty five, almost entirely because champions ran them without us. Findings caught in code review rather than by a scanner rose steadily, and no amount of tooling spend buys you that.

And then the one that actually matters. Renewal rate. When people renew, the role is worth having. When they quietly do not, something upstream is broken and you have about one quarter to find it before the whole thing goes back to being a Slack channel with stickers in it.

The short version

Champions programmes fail because they ask engineers to do unfunded, uninteresting work for the benefit of somebody else's metrics, and then act betrayed when it does not stick.

They work when the time is funded in the sprint, the role carries real authority and real perks, the toil stays with the security team, and doing it is visibly good for the champion's career.

Fix the incentives and the security content is the easy part.

Skip the incentives and the best curriculum in the world will not save you. It will just be very well formatted, like all the other things we leave behind.

---

// Elusive Thoughts // written from the in-house chair, not the consultant one // securityhorror.blogspot.com

Figures are from one estate over roughly two years. Treat as direction, not decimals. Analysis and commentary are my own.

#AppSec #SecurityChampions #SecurityLeadership #DevSecOps #EngineeringCulture

Running on this device Post you do not have 4000 vulnerabilities · HTML

You Do Not Have Four Thousand Vulnerabilities

// elusive thoughts // appsec // sca // the number is lying to you

Somebody turned the dependency scanner on last quarter and it came back with four thousand one hundred findings. Six hundred of them critical or high. The dashboard went red and has stayed red ever since, the way a smoke alarm with a flat battery stays red, and everybody has stopped looking at it for exactly the same reason.

Here is the arithmetic nobody does out loud. Six hundred high severity findings, at a generous two hours each to investigate, patch, test and ship, is twelve hundred engineering hours. Thirty engineer weeks. You are asking a company to spend most of a person year fixing things you have no evidence affect you at all. And next month the scanner will find more.

The engineers worked this out before you did. They triaged the first fifty, found that most were irrelevant, and quietly concluded that the tool cries wolf. That conclusion is correct. It is also permanent. You did not buy security. You bought a very expensive way to teach four hundred developers to ignore a security tool, and they learned it fast, because they are good at their jobs.

The scanner is answering a question you did not ask

A software composition analysis tool reads your lockfile, resolves the tree, matches every package and version against a vulnerability database, and reports. That is the whole algorithm. Notice what is missing from it. At no point does it check whether your code can actually reach the vulnerable function.

A CVE is not a property of a package. It is a property of a specific function inside that package. The remote code execution in a library's XML parser is your problem only if you parse XML with it. If you pulled the library in for a date helper and never touch the parser, that vulnerable code is sitting in node_modules like an unloaded gun in a locked safe in a room you have never entered. The scanner sees a version string and it fires. It cannot tell the difference, and it was never built to.

the pattern

Every finding your scanner reports is the answer to "is this vulnerable code present." The question you actually need answered is "can this vulnerable code run." Those are different questions and the gap between them is where your entire backlog lives.

What the number looks like when you measure it properly

Reachability analysis builds a call graph. It starts at your real entry points, the HTTP handlers, the CLI mains, the queue consumers, and it walks outward through your code, into your direct dependencies, and on into the transitive ones. Then for each CVE it asks one question: is there a path from an entry point to the vulnerable symbol.

If yes, the finding is real. If no, the vulnerable code is present but dead.

Run that across an estate and the numbers stop being frightening and start being useful.

612

high and critical findings reported by the scanner

178

actually reachable from any entry point

~70%

noise, with no path to the vulnerable code at all

Seventy percent is not a rounding error. Seventy percent is the entire signal to noise problem, and it is the difference between a security team that can hold an SLA and a security team that is a running joke in the engineering all hands.

Three things vendors all call reachability

They do not mean the same thing. Know which one you are being sold.

Manifest level. Is the package imported anywhere at all, or is it just squatting in the lockfile. Cheap, fast, works in any language, and it will clear out your dev dependencies and your orphans. It is also a low bar, because "we import the library" is true of almost everything you care about. Expect twenty to thirty percent off the pile.

Function level, static. The real thing. Build the call graph, check the vulnerable symbol, not just the package. This is where the value is and it is also where the engineering gets hard, because it needs per language analysis, it needs a vulnerability database annotated with vulnerable symbols, and it fights badly with reflection, dynamic imports and dependency injection. Go and Java behave. Python and JavaScript do not. Expect sixty to eighty percent off the pile, which is the number that changes your life.

Runtime. Instrument the app, watch what actually loads and executes. Highest fidelity, no static guesswork, and it accounts for configuration for free. The catch is that it tells you what ran, not what could run. The vulnerable admin endpoint nobody hit this week looks unreachable. It is not unreachable. It is unvisited. Use it to confirm a finding, never to dismiss one.

The question that separates a real vendor from a reskinned scanner

Ignore the marketing number. Everybody claims ninety percent noise reduction. Ask this instead, and ask it in the demo, out loud.

the test

What percentage of the CVEs in your database have symbol level annotations, and what do you do when you cannot statically resolve a call.

The first half matters because reachability is only possible for CVEs where somebody bothered to identify the vulnerable function. If the honest answer is forty percent, then sixty percent of your findings fall back to package level analysis no matter how good the call graph engine is. That is the dirty secret of this entire product category and most buyers never ask.

The second half matters more. A tool that marks a genuinely reachable vulnerability as unreachable is far more dangerous than the noise you started with, because now you are confidently ignoring a real bug and you have a report that says you were right to. The correct answer is some version of "we treat unresolvable dynamic dispatch as potentially reachable and we fail open." If a vendor tells you they statically resolve all dynamic dispatch, they are either lying to you or they have not thought about it. Both are disqualifying and you should end the call.

What I would actually do

Measure in the dark for a month. Turn reachability on and change nothing about how findings are handled. You are building a case, not a process. What you want at the end is one sentence you can say to a VP: we have six hundred and twelve high findings, one hundred and seventy eight are reachable, and we have been asking engineering to spend thirty weeks on four hundred and thirty four issues that cannot be exploited. Nobody argues with that sentence.

Split the queue, do not shrink it. Reachable findings get an SLA and a ticket. Everything else goes to a dashboard with no SLA and no ticket, and it is re evaluated on every single build. Say the words carefully and say them often: not currently reachable is not the same as not vulnerable. The code is still in the image. The day somebody adds a call to that function, your next build must promote it into the real queue automatically. If it does not promote automatically then you have not built noise reduction, you have built a way to lose track of vulnerabilities, and one day you will explain that to an auditor.

Demand the path, not the verdict. If the tool cannot show handler to service to library to CVE, engineers will not believe it, and a finding a developer does not believe is a finding that does not get fixed. Explainability is not a nice to have here. It is the whole adoption mechanism.

Put it on merge, not on the pull request. Full call graph analysis is expensive. Add eight minutes to every PR build and engineers will route around it and you are back where you started. Fast package check on the PR, full analysis nightly.

Then go after the roots. With the noise gone the patterns finally become visible. One ancient internal SDK generating half your reachable findings. Three services on an end of life runtime. Those are platform fixes. Upgrade once, close a hundred findings, permanently. You could never see them underneath four thousand one hundred rows and that, not the smaller number, is the actual prize.

The part that is still your job

Reachable is not exploitable. A reachable function is not necessarily reachable with attacker controlled input. Reachability takes you from four thousand to two hundred. It does not take you from two hundred to the twelve a competent attacker could really use. That last mile is human judgment and there is no product for it, which is presumably why nobody sells one.

And it does not save you everywhere. Compiled binaries, vendored code, base image OS packages that live entirely outside the call graph. Keep package level scanning as the floor. Reachability is a prioritisation layer sitting on top of it, not a replacement for it.

But it is the difference between a queue engineering can clear and a queue engineering has already given up on.

You do not have a vulnerability problem. You have a prioritisation problem wearing a vulnerability problem's clothes, and you have been paying a licence fee to keep the costume on.

Measure it before you argue about it. The seventy percent will make the argument for you.

---

// Elusive Thoughts // the number was never the point // securityhorror.blogspot.com

SOURCES // Figures drawn from a roughly 140 service estate. Cross checked against published SCA noise studies. Treat as direction, not decimals. Analysis and commentary are my own.

#AppSec #SCA #SupplyChain #VulnerabilityManagement #DevSecOps #Reachability