Pentesting the Agent: Where AI Workflows Actually Break

// elusive thoughts // ai security // agentic // stop testing the model, test the wiring

Most people who say they are pentesting an AI feature are testing the wrong thing. They open the chat window, they type ignore all previous instructions, they get the model to say something rude, they screenshot it, and they file a finding about prompt injection. That is not a pentest. That is a party trick, and it teaches the developers that AI security is about naughty words, which is exactly the lesson you do not want them to learn.

The interesting target was never the model. It is the wiring around it. Somewhere between the user and the language model there is a system that retrieves documents, calls tools, holds credentials, and renders output into a browser. That system was assembled quickly, by people under deadline, on top of a component nobody in the building fully understands. That is where you break in. The model is just the confused employee you socially engineer to do it.

The mental model that makes findings fall out

Draw the thing as it actually is. Not a magic oracle. An untrusted, highly persuadable component that sits inside your trust boundary and holds credentials. The moment you draw it that way, the whole test plan writes itself, because you have seen this shape before. A confused deputy with a keyring is a thirty year old problem. It just arrived this year wearing a hoodie and calling itself an assistant.

the pattern

Assume the injection succeeds. Do not spend the engagement trying to prove you can inject. You can. Everybody can. Spend it proving what the injection reaches. Prompt injection is not the vulnerability. It is the unauthenticated foothold. The vulnerability is everything the foothold can touch.

Road one: the model has more access than the user driving it

The first thing to enumerate on any agent is whose authority it acts with. Nine times out of ten the answer is a service account, because that was the easy way to ship, and the service account can read across tenants because scoping it properly was a later ticket that never came.

Now you have your attack. You do not need to break the model. You need to get untrusted text in front of it and let it use the badge it already holds. A document you upload. A calendar invite. A support ticket. A web page the agent is asked to summarise. Any channel where content you control becomes content the model reads is an instruction channel, and the model will happily act on it with permissions the human sending the request never had.

The test is simple. Plant an instruction in a document that only your low privilege user can see, ask the agent an innocent question, and watch whether it reaches data your user cannot. If it does, you did not find an AI bug. You found a privilege escalation that happens to have a chatbot in the middle, and it should be written up with exactly that severity.

Road two: the retrieval layer never checked permissions

Retrieval augmented generation is where the quiet breaches live. Somebody embedded the company's documents into a vector store so the assistant could answer questions about them. Ask whether the retrieval query respects document level permissions, and the honest answer is often that the permission check happens at display time, after the content has already been pulled into the model's context.

Think about what that means. The model has already read the document the user is not allowed to see. The access control is a curtain drawn after the safe was opened. And the model does not honour curtains. Ask it the right question and it will summarise for you, cheerfully, the contents of files your account has no rights to, because as far as the retrieval layer was concerned every embedding in the index was fair game.

To test it you need two users and one shared store. Index a secret as the privileged user. Log in as the other one. Ask around the secret rather than for it. If the summary contains something your account cannot open directly, the permission model lives in the wrong place and you have a real data exposure.

Road three: the output is rendered, and rendering executes

This is the road pentesters miss because it does not look like AI at all. It looks like markdown working as intended.

The model returns markdown. The frontend renders it. Markdown supports images. Images have URLs. URLs carry query strings. Put those four facts together and you have a data exfiltration channel that fires with no click.

An instruction buried in a retrieved document tells the model to end its answer with an image whose address is an attacker server followed by a summary of the conversation. The frontend renders the image tag. The browser fetches it automatically, and the fetch carries the stolen content in the URL. Nobody clicked anything. The user watched a helpful answer appear and a beacon left the building underneath it.

field note

The best AI finding I have ever written was old fashioned cross site scripting thinking pointed at a new output channel. No clever jailbreak. Just markdown, an image tag, and a frontend that trusted the model's output the way we all learned twenty years ago never to trust user input. The model is user input now. Treat everything it emits as attacker controlled, because with one injected instruction it is.

Road four: the tools have verbs they should not

Once an agent can call tools, enumerate them like an API, because that is what they are. For each one ask two questions. What does it do, and what does it do with attacker supplied arguments.

The dangerous pattern is a tool with a side effect and no confirmation. Send email. Delete record. Post to channel. Make outbound request. If the model can call those and the model can be steered by injected text, then the injected text can send the email, delete the record, and make the request to wherever it likes. The classic escalation is a tool that fetches a URL, because that is your exfiltration path and your server side request forgery in one, handed to you by the feature itself.

The test plan is boring and effective. Scope every tool. Inject an instruction that calls the most dangerous one with arguments you control. See if a human confirmation stands between the model and the action. If nothing does, the blast radius of a single planted sentence is the full set of verbs you just enumerated.

Road five: nobody can tell you what it did

End every agent engagement with one request. Show me the logs of what the model actually did during my testing. Which tools it called, with which arguments, against whose data, and what came back.

Most of the time the log says the assistant was invoked and nothing else. That is a finding on its own, and a serious one, because it means that when a real attacker walks these same roads the incident response will be a shrug and a sentence that begins we think it did something. An agent action is a privileged action. If you would log an admin deleting a user, you log the model deleting a user, with the same fields, or you are flying blind on purpose.

What to hand the developers

Not a jailbreak transcript. Give them the boundary decisions, because those are the fixes.

Collapse the model's identity into the caller's. Every tool call runs with the user's token through the same authorisation layer as the rest of the product. The model may ask for anything. The authz layer decides what it gets.

Move the permission check to query time, inside the retrieval store, before anything reaches the context window. Slower and correct beats fast and leaking.

Treat model output as untrusted. Allowlist rendered image and link domains. No auto loading of external resources. The same discipline you would apply to any user generated content, because that is now what it is.

Put a human in front of every side effect, and allowlist outbound calls from the tool layer.

Log every tool invocation as a first class audit event. Actor, tool, arguments, authorisation decision, result size. The same as any other privileged action, because it is one.

The reframe

The OWASP list for large language models is a fine place to start and it will not save you, because every finding on these five roads is a classic problem in an unfamiliar costume. Confused deputy. Missing authz on the read path. Output encoding. Server side request forgery. Insufficient logging. We have known how to test all of these since before the model existed.

Stop trying to out argue the model. You will lose, it has read more than you and it never gets tired. Draw it as what it is, a persuadable insider with a keyring, and test the keyring. The magic is not in the model. It never was. The magic was always in how much access somebody handed it on a Friday afternoon to make the demo work.

---

// Elusive Thoughts // the model is the mark, not the target // securityhorror.blogspot.com

Techniques described are for authorised testing of systems you own or have permission to assess. Analysis and commentary are my own.

#AIsecurity #LLMhacking #AgenticAI #PromptInjection #RAG #AppSec #PenTest #OWASP

Your Champions Programme Is Unpaid Overtime With a Sticker

// elusive thoughts // appsec // culture // fund it or do not run it

There is a Slack channel at your company called something like security champions. It was created with a lot of energy. There was a kickoff, there were stickers, there was a deck with a diagram showing six security engineers in the middle and forty developers arranged around them like a solar system. Somebody said the word force multiplier and nobody laughed.

Scroll it now. The last message is four months old. It is from you.

I have built this programme twice. The first one died exactly like that. The second one is still running, and the difference between them had almost nothing to do with security and almost everything to do with money and promotion, which is a sentence nobody wants to hear at a security conference and is nevertheless the whole of what I have to say.

The five ways it dies

The volunteer trap. You sent an email asking who wants to do it. Hands went up, good hands, people who genuinely care. Then it was a Tuesday, the team was two weeks behind on the quarterly commitment, and their manager, who never agreed to any of this and whose bonus is tied to shipping, asked what they were working on. Security champion work is not a sprint item. It is not on the roadmap. It is not what they will be assessed on in six months. The channel goes quiet in about eight weeks and everybody is too polite to say why.

The dumping ground. Now that a champion exists, you have somewhere to put things. Scanner findings for that team, send them to the champion. Security questionnaire needs a technical answer, champion. Training compliance chase, champion. Within one quarter the role has become "the person who does the security team's admin for free" and everybody can see it. The tell is easy to spot. Look at what your champions actually did last month. If nearly all of it was reactive, and nearly none of it made their own team's software better, you have built a punishment and attached a badge to it.

The training programme that changed nothing. You built a curriculum. Secure coding, the OWASP Top 10, a CTF, maybe a cert. Everybody completed it. Everybody learned something. Nothing changed, because knowledge was never the bottleneck. The champion who now knows what an IDOR is still has no time to review pull requests, no authority to block a bad design, and no standing to tell their tech lead the auth model is wrong. You handed them a map and no vehicle. Training keeps happening anyway, because completion rates are easy to report and they make a very good slide.

The one hero. One champion is genuinely excellent. They find real bugs, they push back on bad designs, other teams start asking for them. So they get more work, and more, and their own roadmap slips, and their manager notices, and their review says needs to focus. Then they get promoted out, or they leave, and the programme's entire output, which was functionally one person, goes to zero in a single Friday. If your results are concentrated in one or two people you do not have a programme. You have a dependency, and you should be losing sleep over it.

The cargo cult. You watched the Google talk, or the Netflix one, and you copied the structure. Guild, monthly sync, maturity model, RACI. None of it is connected to anything an engineer at your company experiences on a Tuesday. The meeting happens because it is in the calendar. Attendance falls ten percent a month until it is you and one loyal person who feels bad for you. You copied the artefact and not the conditions that produced it. At those companies the incentives and the executive backing came first and the org structure was the last thing they added. You did it in reverse and wondered why it did not take.

The thing that actually worked

Before we named a single champion, we went to engineering leadership with an ask so boring it is almost embarrassing to write down.

Ten percent of one engineer's time per team. Four hours a week. Named in the sprint, on the board, planned around, in exactly the same place as every other commitment the team makes.

Not as time allows. Not twenty percent time. Not a spiritual commitment made in a kickoff meeting. A line item.

And if a manager would not fund the four hours, that team did not get a champion. Not once. Not even when somebody great wanted it. Not even when it made our coverage numbers look bad in front of the CISO, which it did, for two quarters.

field note

That refusal was the single highest leverage decision I have made in this job. It converted security champion from a favour into a role. Favours evaporate when the quarter gets tight. Roles have budgets, and things with budgets get defended. It also handed me an honest metric I did not have before, teams with a funded champion against teams without, and the gap between those two columns is a conversation with a VP rather than a complaint in a retro.

Define the role by what they own, not what they attend

Four things. That is the list. Nothing else belongs to them.

They triage their own team's findings, and they have the authority to close something as a false positive without asking us. That authority is real and giving it away is the point. It is what makes the role feel like a promotion instead of a chore.

They run the threat model for their team's new services. Forty five minutes, four people, a whiteboard. We taught the format, sat in on the first two, and then got out of the way, which was harder than it sounds.

They review the pull requests that matter. Auth changes, new endpoints, crypto, anything crossing a boundary. Not every PR. The ones that count, routed automatically by a CODEOWNERS rule so that it does not depend on anybody remembering.

They are the escalation path to us, and not the escape valve for us. If it is beyond them it comes to AppSec fast and there is no shame attached to it arriving.

Now look at what is not on that list. Chasing training completion. Filling out customer questionnaires. Writing policy. Being the compliance liaison. We kept every unpleasant administrative task, deliberately, forever. The champions get the interesting work and the security team keeps the toil. Reverse that and the programme is dead inside a quarter and you will not even be able to say when it happened.

Put it on the promotion ladder, in writing

We spent three months in unglamorous meetings with HR and engineering directors to get security ownership named explicitly in the career framework, as evidence toward the technical leadership beyond your immediate team criterion that every senior and staff track already has.

That was worth more than every piece of security content we have ever produced.

Because now when a champion's manager asks why they are spending four hours a week on this, the answer is not because I like security. The answer is because it is how I am demonstrating staff level scope, and here is the framework language that says so.

Nothing sustains discretionary effort like it being the fastest route to a promotion. That is not cynicism. That is just how organisations work, and pretending otherwise has killed more security programmes than any attacker.

Give them things nobody else has

Champions got write access to the security tooling config for their own repos, so they could tune rules and thresholds themselves rather than filing a request with us. A private channel where the answer arrives in minutes. Early access to new tooling and a real vote on what we bought. A conference budget line and an explicit expectation that they use it.

The message underneath all of that is the recruiting strategy. This role gives you access and capability you cannot get any other way. Champions should be visibly better equipped than their peers. People should want the job, and wanting the job is the only recruitment mechanism that scales past the first cohort.

Rotate them on purpose

A one year term, then renew or hand off, with a deliberate two month overlap.

This prevents the one hero failure. It turns a champion leaving into a scheduled event instead of a crisis. And over time it spreads security knowledge far wider than a permanent role ever could. After three years you do not have twenty champions. You have twenty champions and forty former champions who still read a diff carefully out of habit, and that second number is the one that is quietly making your software better.

Rotation feels like it weakens the programme. It is the thing that lets it survive contact with reality.

Measure the right things

Not number of champions. Not training completion. Those measure activity, and activity is what dying programmes report right up until the week they are cancelled.

Time to triage on new findings went from eleven days to two, which is the clearest signal that real work is happening closer to the code. New services with a threat model at design time went from twenty percent to eighty five, almost entirely because champions ran them without us. Findings caught in code review rather than by a scanner rose steadily, and no amount of tooling spend buys you that.

And then the one that actually matters. Renewal rate. When people renew, the role is worth having. When they quietly do not, something upstream is broken and you have about one quarter to find it before the whole thing goes back to being a Slack channel with stickers in it.

The short version

Champions programmes fail because they ask engineers to do unfunded, uninteresting work for the benefit of somebody else's metrics, and then act betrayed when it does not stick.

They work when the time is funded in the sprint, the role carries real authority and real perks, the toil stays with the security team, and doing it is visibly good for the champion's career.

Fix the incentives and the security content is the easy part.

Skip the incentives and the best curriculum in the world will not save you. It will just be very well formatted, like all the other things we leave behind.

---

// Elusive Thoughts // written from the in-house chair, not the consultant one // securityhorror.blogspot.com

Figures are from one estate over roughly two years. Treat as direction, not decimals. Analysis and commentary are my own.

#AppSec #SecurityChampions #SecurityLeadership #DevSecOps #EngineeringCulture

Running on this device Post you do not have 4000 vulnerabilities · HTML

You Do Not Have Four Thousand Vulnerabilities

// elusive thoughts // appsec // sca // the number is lying to you

Somebody turned the dependency scanner on last quarter and it came back with four thousand one hundred findings. Six hundred of them critical or high. The dashboard went red and has stayed red ever since, the way a smoke alarm with a flat battery stays red, and everybody has stopped looking at it for exactly the same reason.

Here is the arithmetic nobody does out loud. Six hundred high severity findings, at a generous two hours each to investigate, patch, test and ship, is twelve hundred engineering hours. Thirty engineer weeks. You are asking a company to spend most of a person year fixing things you have no evidence affect you at all. And next month the scanner will find more.

The engineers worked this out before you did. They triaged the first fifty, found that most were irrelevant, and quietly concluded that the tool cries wolf. That conclusion is correct. It is also permanent. You did not buy security. You bought a very expensive way to teach four hundred developers to ignore a security tool, and they learned it fast, because they are good at their jobs.

The scanner is answering a question you did not ask

A software composition analysis tool reads your lockfile, resolves the tree, matches every package and version against a vulnerability database, and reports. That is the whole algorithm. Notice what is missing from it. At no point does it check whether your code can actually reach the vulnerable function.

A CVE is not a property of a package. It is a property of a specific function inside that package. The remote code execution in a library's XML parser is your problem only if you parse XML with it. If you pulled the library in for a date helper and never touch the parser, that vulnerable code is sitting in node_modules like an unloaded gun in a locked safe in a room you have never entered. The scanner sees a version string and it fires. It cannot tell the difference, and it was never built to.

the pattern

Every finding your scanner reports is the answer to "is this vulnerable code present." The question you actually need answered is "can this vulnerable code run." Those are different questions and the gap between them is where your entire backlog lives.

What the number looks like when you measure it properly

Reachability analysis builds a call graph. It starts at your real entry points, the HTTP handlers, the CLI mains, the queue consumers, and it walks outward through your code, into your direct dependencies, and on into the transitive ones. Then for each CVE it asks one question: is there a path from an entry point to the vulnerable symbol.

If yes, the finding is real. If no, the vulnerable code is present but dead.

Run that across an estate and the numbers stop being frightening and start being useful.

612

high and critical findings reported by the scanner

178

actually reachable from any entry point

~70%

noise, with no path to the vulnerable code at all

Seventy percent is not a rounding error. Seventy percent is the entire signal to noise problem, and it is the difference between a security team that can hold an SLA and a security team that is a running joke in the engineering all hands.

Three things vendors all call reachability

They do not mean the same thing. Know which one you are being sold.

Manifest level. Is the package imported anywhere at all, or is it just squatting in the lockfile. Cheap, fast, works in any language, and it will clear out your dev dependencies and your orphans. It is also a low bar, because "we import the library" is true of almost everything you care about. Expect twenty to thirty percent off the pile.

Function level, static. The real thing. Build the call graph, check the vulnerable symbol, not just the package. This is where the value is and it is also where the engineering gets hard, because it needs per language analysis, it needs a vulnerability database annotated with vulnerable symbols, and it fights badly with reflection, dynamic imports and dependency injection. Go and Java behave. Python and JavaScript do not. Expect sixty to eighty percent off the pile, which is the number that changes your life.

Runtime. Instrument the app, watch what actually loads and executes. Highest fidelity, no static guesswork, and it accounts for configuration for free. The catch is that it tells you what ran, not what could run. The vulnerable admin endpoint nobody hit this week looks unreachable. It is not unreachable. It is unvisited. Use it to confirm a finding, never to dismiss one.

The question that separates a real vendor from a reskinned scanner

Ignore the marketing number. Everybody claims ninety percent noise reduction. Ask this instead, and ask it in the demo, out loud.

the test

What percentage of the CVEs in your database have symbol level annotations, and what do you do when you cannot statically resolve a call.

The first half matters because reachability is only possible for CVEs where somebody bothered to identify the vulnerable function. If the honest answer is forty percent, then sixty percent of your findings fall back to package level analysis no matter how good the call graph engine is. That is the dirty secret of this entire product category and most buyers never ask.

The second half matters more. A tool that marks a genuinely reachable vulnerability as unreachable is far more dangerous than the noise you started with, because now you are confidently ignoring a real bug and you have a report that says you were right to. The correct answer is some version of "we treat unresolvable dynamic dispatch as potentially reachable and we fail open." If a vendor tells you they statically resolve all dynamic dispatch, they are either lying to you or they have not thought about it. Both are disqualifying and you should end the call.

What I would actually do

Measure in the dark for a month. Turn reachability on and change nothing about how findings are handled. You are building a case, not a process. What you want at the end is one sentence you can say to a VP: we have six hundred and twelve high findings, one hundred and seventy eight are reachable, and we have been asking engineering to spend thirty weeks on four hundred and thirty four issues that cannot be exploited. Nobody argues with that sentence.

Split the queue, do not shrink it. Reachable findings get an SLA and a ticket. Everything else goes to a dashboard with no SLA and no ticket, and it is re evaluated on every single build. Say the words carefully and say them often: not currently reachable is not the same as not vulnerable. The code is still in the image. The day somebody adds a call to that function, your next build must promote it into the real queue automatically. If it does not promote automatically then you have not built noise reduction, you have built a way to lose track of vulnerabilities, and one day you will explain that to an auditor.

Demand the path, not the verdict. If the tool cannot show handler to service to library to CVE, engineers will not believe it, and a finding a developer does not believe is a finding that does not get fixed. Explainability is not a nice to have here. It is the whole adoption mechanism.

Put it on merge, not on the pull request. Full call graph analysis is expensive. Add eight minutes to every PR build and engineers will route around it and you are back where you started. Fast package check on the PR, full analysis nightly.

Then go after the roots. With the noise gone the patterns finally become visible. One ancient internal SDK generating half your reachable findings. Three services on an end of life runtime. Those are platform fixes. Upgrade once, close a hundred findings, permanently. You could never see them underneath four thousand one hundred rows and that, not the smaller number, is the actual prize.

The part that is still your job

Reachable is not exploitable. A reachable function is not necessarily reachable with attacker controlled input. Reachability takes you from four thousand to two hundred. It does not take you from two hundred to the twelve a competent attacker could really use. That last mile is human judgment and there is no product for it, which is presumably why nobody sells one.

And it does not save you everywhere. Compiled binaries, vendored code, base image OS packages that live entirely outside the call graph. Keep package level scanning as the floor. Reachability is a prioritisation layer sitting on top of it, not a replacement for it.

But it is the difference between a queue engineering can clear and a queue engineering has already given up on.

You do not have a vulnerability problem. You have a prioritisation problem wearing a vulnerability problem's clothes, and you have been paying a licence fee to keep the costume on.

Measure it before you argue about it. The seventy percent will make the argument for you.

---

// Elusive Thoughts // the number was never the point // securityhorror.blogspot.com

SOURCES // Figures drawn from a roughly 140 service estate. Cross checked against published SCA noise studies. Treat as direction, not decimals. Analysis and commentary are my own.

#AppSec #SCA #SupplyChain #VulnerabilityManagement #DevSecOps #Reachability

A Threat Model Developers Will Actually Use

// appsec // threat modeling // kill the 40-page pdf

Somewhere on a shared drive at almost every company is a folder of threat models. Forty pages each. Beautiful diagrams. A STRIDE matrix that took someone two days to fill in. Every one of them was opened exactly twice. Once when it was written, once when an auditor asked for evidence that threat modeling happens. Between those two opens, the system it described shipped, changed shape four times, and grew an entire new attack surface the document never mentioned.

That folder is a graveyard. The headstones are very well formatted. And it is the single biggest reason developers think threat modeling is security theatre, because the version they were shown is theatre. A document produced for an audience of one auditor, read by no engineer, describing a system that no longer exists.

I want to talk about the other kind. The threat model a developer opens on purpose, mid-change, because it helps them ship the right thing. It is smaller than you think, it lives somewhere you might not expect, and the format barely matters.

Why threat models die

Three causes of death, and they are almost always the same three.

It is too big. A forty-page model of an entire platform is obsolete the day it is signed. Systems do not change at platform granularity. They change one pull request at a time. A model scoped to the whole system can never keep up with a system that changes in small pieces, so it rots, and a rotting model is worse than none because it lies with authority.

It lives in the wrong place. If the threat model is a Confluence page or a Visio file in a drive, it is outside the developer's loop. Developers live in the repo, the PR, the IDE, the ticket. Anything outside that loop is a context switch, and context switches under deadline lose every time. The model is not lazy. It is just geographically inconvenient.

It produces a document, not a decision. This is the deep one. Most threat models output a description of risks. A developer cannot act on a description. They can act on a decision: add auth to this endpoint, do not log this field, this service must not be reachable from the internet. If your threat model ends in prose instead of a short list of concrete changes, you produced a report, and reports go in the graveyard.

the test

A threat model is useful if, and only if, a developer can read it in five minutes and come away with a small list of things they will now build differently. Everything that does not serve that outcome is decoration.

What "usable" actually looks like

Flip the three causes of death and you get the design spec for free.

Scoped to a change, not a system. You do not threat model "the payments platform." You threat model "the new endpoint that lets a user move money between their own pots." Small enough to finish in one sitting. Small enough to stay true, because it is attached to a single change that either shipped or did not.

Lives where the work lives. A markdown file in the repo, next to the code. A section in the design doc that already exists for any non-trivial change. A comment thread on the PR. The model should be reachable without leaving the place the developer already is. If they have to open a second tab, you have already lost most of them.

Ends in a checklist of decisions. The output is not "here are the STRIDE categories we considered." The output is a handful of concrete, testable statements about what the code will now do. Those statements become acceptance criteria. They become test cases. They become review comments. They are load-bearing, not archival.

The method, kept deliberately small

You can run a genuinely useful threat model on a change with four questions. No matrix, no tooling, no two-day workshop. Sit with the one or two engineers who own the change and ask, in order:

1. What are we building, in one diagram? Not a formal DFD. A napkin sketch of the boxes, the arrows between them, and where the data goes. The only rule: draw the trust boundaries. Where does data cross from something you control to something you do not, or from less-trusted to more-trusted? Every interesting bug lives on a boundary line.
2. What would an attacker want here, and what are they allowed to touch? Money, personal data, a privilege, a foothold. And critically, who is the attacker? An anonymous internet user, a logged-in customer poking at another customer's data, a compromised internal service. The threat model for "external user" and "malicious authenticated user" are different models and most teams only ever do the first.
3. How could each boundary crossing go wrong? Walk every arrow that crosses a trust boundary. For each one: can it be spoofed, tampered with, abused to read something it should not, abused to do something it should not. This is STRIDE if you want a name for it, but you do not need the acronym in the room. You need the engineers looking at their own arrows and getting uncomfortable.
4. What are we going to do about it? This is the only question whose answer you keep. Each real risk becomes one of three things: a change we make now, a risk we knowingly accept and write down, or a thing we explicitly decide is out of scope. Three buckets. No prose.

That is the whole method. Thirty to sixty minutes for a normal change. The artifact it produces is the answers to question four, and nothing else needs to survive.

What the artifact looks like

Here is the entire output for a real-shaped example, the money-between-pots endpoint, as a markdown block that lives in the PR:

## Threat model: internal pot transfer (PR #2241)

Boundary crossings:
- client -> transfer API  (untrusted -> trusted)
- transfer API -> ledger  (trusted -> trusted, money moves here)

Decisions:
[x] enforce that both pots belong to the authenticated user
      (authz, not just authn) -> test: transfer to a pot you
      do not own returns 403
[x] make the transfer idempotent on a client key
      -> replaying the same request moves money once, not twice
[x] do not log the amount or pot IDs at info level (PII + abuse)
[ ] ACCEPTED RISK: no per-user rate limit at launch.
      Owner: payments. Revisit before raising transfer caps.
OUT OF SCOPE: cross-user transfers (not built in this change)

That is it. A developer reads it in two minutes. Three of those lines are now test cases. One is a written, owned risk acceptance instead of a thing nobody decided. One is an explicit scope boundary so the next person does not assume it was missed. No auditor will be thrilled by its length. Every engineer who touches this code will actually use it.

Make it a habit, not an event

The reason the four-question version wins long term is not that it is better security. A two-day workshop finds more, in theory. The four-question version wins because it is cheap enough to run every time, and a cheap thing done every time beats a thorough thing done never.

Threat modeling as an event, a quarterly ceremony with a facilitator and a meeting room, models the system at one frozen moment and then watches reality drift away from it. Threat modeling as a habit, a thirty-minute section in the design doc for any change that touches a trust boundary, moves with the system because it is part of how the system gets built. The first is an artifact. The second is a reflex. You want the reflex.

The way you get the reflex is by making it small enough that nobody can argue it is not worth the time, and by putting it where they already work so it costs them no context switch. Drop a four-question template into the repo as THREAT_MODEL.md next to the design-doc template. Add one line to the PR checklist: does this change cross a trust boundary, and if so, link the model. That is the entire rollout. No platform purchase, no workshop calendar.

field note

The best threat models I see in-house are three paragraphs of markdown in a PR, written by the developer, with two comments from me. The worst are forty-page PDFs written by security in isolation. The difference in security value is not subtle, and it runs the opposite way to the page count.

Where the heavy version still earns its place

To be fair to the forty-page model: there is a place for depth. A new platform, a major architectural decision, a system that moves serious money or holds serious data deserves a real, deep, deliberate threat model with the full method and the diagrams and the time. That is a handful of times a year, on the things that genuinely warrant it.

The mistake is using that heavyweight format for everything, because then you either do it rarely, on the big things only, and leave every normal change unmodelled, or you try to do it on everything and burn out in a month. Match the weight of the model to the weight of the change. Big decision, deep model, run it like the serious thing it is. Normal change crossing a boundary, four questions and a markdown block. Trivial change touching nothing sensitive, no model at all, and be honest that this is most changes.

Calibrating that is the actual skill. Not filling in the matrix. Knowing which changes deserve which depth, and refusing to spend a forty-page budget on a four-question problem. Do that, and threat modeling stops being the thing developers dread and the folder nobody opens. It becomes a thirty-minute reflex that quietly catches the bugs that would otherwise have been a pentest finding six weeks too late.

---

// Elusive Thoughts // less paper, more decisions // securityhorror.blogspot.com

Shift-Left Is Org Design Wearing a Vendor Badge

// appsec // culture // the part nobody sells you

Every shift-left pitch I have sat through follows the same arc. A scanner plugs into CI. A dashboard turns red. Developers fix things earlier. Risk slides down and to the left like a stock chart nobody questions. Buy the tool, get the outcome. It is clean, it ships quarterly, and it is mostly a lie.

Not because the tools are bad. Some of them are excellent. It is a lie because the slide deck quietly swaps the cause and the effect. Shift-left is not a thing you install. It is a thing your organisation already is, or already is not. The tool is the last 10 percent. The other 90 percent is who owns what, who gets paged, who can say no, and how fast a developer finds out they made a mistake. That is org design, and you cannot buy it on a per-seat licence.

What the word actually means once you strip the paint

Strip the marketing and shift-left says one thing: move the security decision closer to the moment the decision is made. That moment is not the pull request. It is earlier. It is a developer choosing a library, sketching a data flow on a whiteboard, deciding whether a service needs to talk to another service at all. The bug you want to kill was born in that choice, not in the diff.

So the real question shift-left asks is not where do we put the scanner. It is who is in the room when the choice gets made, and do they have what they need to choose well. A scanner in CI is downstream of every interesting answer to that question. It catches what survived the choice. Useful, but it is a net under the tightrope, not the tightrope.

Why tooling-first cargo-cults itself to death

Here is the failure mode I have watched play out more than once, from both chairs, consultant and in-house.

Security buys a scanner. Security wires it into the pipeline. The first full run produces 4,000 findings. Nobody triaged the baseline, so 3,600 of them are noise, dead code, test fixtures, dependencies that are technically vulnerable in a path that does not exist in production. The pipeline goes red. Developers cannot ship. Developers do the only rational thing a human under a deadline does: they find the bypass. A skip label. A nightly job instead of a blocking gate. A Slack message to the one person with admin who will wave it through.

Now you have spent budget to teach your entire engineering org that the security gate is a thing you route around. That lesson is durable. It outlives the tool. The next tool you bring in inherits the reputation of the last one, and you wonder why adoption is a fight every single time.

the pattern

A tool dropped into an org that is not built to absorb it does not raise the security baseline. It raises the noise floor and trains people to ignore you. The tool worked perfectly. The org ate it.

None of that is a tooling problem. You can swap the vendor and reproduce the exact same wreck. It is an ownership problem, an incentive problem, and a feedback-latency problem wearing a tooling costume.

The primitives that actually move security left

If shift-left is org design, then these are the levers. None of them appear in a product comparison grid.

1. Ownership that survives a reorg

A finding with no owner is not a finding, it is a rumour. The single highest-leverage thing I have done in-house is not picking a scanner. It is making sure every service has a named team that owns its security posture, and that the name is attached to something the team already cares about, not a spreadsheet security maintains in the dark.

If the security backlog lives in a tool only the security team opens, it is dead. Findings have to land in the same backlog where the team plans its sprint, in the same tracker, with the same labels, ranked against the same features. The moment a vuln has to compete with a feature on equal footing in front of the same engineering manager, you have moved security left in the only way that holds. You moved it into the place where prioritisation actually happens.

2. Incentives pointed at the outcome you want

People do what they are measured on. If a team is measured purely on delivery velocity, security is friction, and you are the friction. No amount of lunch-and-learns fixes a misaligned incentive. You can win every developer's heart in the room and lose every one of them the second the quarter clock starts.

The fix is not to make security a KPI nobody believes. It is to make the secure path the fast path. If reaching for the hardened, paved-road service template is genuinely the quickest way to ship, security stops being a tax and becomes the default. That is an engineering-platform investment, not a security-tool purchase. It is built by the platform team, funded by leadership, and your job is to make the case for it in language a delivery lead and a CFO both accept.

3. Decision rights, written down

Who can accept a risk? Who can override a gate? On what authority? If the answer is whoever shouts loudest before the release, you do not have a security program, you have a negotiation that restarts every Friday. Real shift-left needs the boring governance most teams skip: a written risk-acceptance path, an owner for it, and a clear line between what a team can wave through themselves and what has to come up the chain. Boring. Load-bearing.

4. Feedback latency measured in minutes, not weeks

The entire mechanical value of shift-left is shortening the distance between mistake and correction. A pentest that lands findings six weeks after the code shipped has shifted nothing. A SAST rule that flags the dangerous pattern in the IDE, while the developer still has the whole mental model loaded, has shifted everything. Same class of bug. The only variable that changed was latency, and latency is the whole game.

This is the one place tooling genuinely earns its keep, and notice it only works because the org already did the other three. Fast feedback into a team with no ownership and no incentive is just a faster way to get ignored.

So where does the tool go

Last. The tool goes last. Once a service has a named owner, the secure path is the fast path, risk acceptance is a written route instead of a hallway favour, and feedback is fast and tuned, then you bolt on the scanner and it lands on prepared ground. The findings have somewhere to go. Somebody owns them. The baseline was triaged before the gate ever blocked anyone. Adoption is not a fight because you are not asking people to absorb chaos, you are handing them a sharper version of a process they already run.

Do it in the other order and the tool is a liability. I would rather inherit an org with strong ownership and a mediocre scanner than a perfect scanner bolted to an org that routes around it. The first one I can improve in a quarter. The second one taught itself to ignore me, and unlearning that takes years.

field note

The most effective security control I have shipped this year was not a tool. It was getting security findings into the same Jira board, with the same priority scale, as feature work. Zero new software. It changed more behaviour than any scanner I have ever deployed.

What I would actually do on day one

If you handed me a fresh AppSec mandate and a budget, here is the order, and the tool is deliberately not at the top.

• Map ownership first. Every production service to a named team. The gaps you find here are your real risk register, and they cost nothing to find.
• Find the paved road, or build the case for one. What is the fastest way to ship a new service today, and is it secure by default? If the fast path and the safe path are different paths, that is the problem. Fix the road before you fine people for going off it.
• Write down the risk-acceptance path. One page. Who decides, on what authority, recorded where. Kill the Friday negotiation.
• Only then, pick the tool, tune the baseline before it blocks a single build, and wire it into the feedback loop that is closest to the developer you can reach.

Notice that three of those four cost no licence money and do not appear in any vendor demo. That is the tell. Shift-left was always org design. The vendors just found a way to sell you the badge without the work, and the badge does not do the work.

Buy the tool if you want. Buy a good one. But buy it knowing it is the last 10 percent, and that if you skip the 90 percent underneath it, the tool will work flawlessly while your program quietly fails around it.

---

// Elusive Thoughts // written from the in-house chair, not the consultant one // securityhorror.blogspot.com

Nobody Breaks In Anymore

// elusive thoughts // appsec // threat intel // 2026

Read enough 2026 breach reports and something gets embarrassing. The intrusion you picture, the hoodie, the zero-day, the frantic keyboard work, almost never happened. The attacker walked in. Sometimes you left the door open. Sometimes someone sold them the key. Sometimes you imported them yourself and ran their code on your build server with a smile.

Three roads, one destination: your environment. None of them look like hacking. All of them work, and the data from the last six months says they work better than ever.

Road one: the door you left open

Cloud misconfiguration is the breach cause that refuses to die. Not nation-state magic. Not a novel exploit chain. A storage bucket set to public. An IAM role with *:* because someone needed it to ship on a Friday. A security group open to 0.0.0.0/0 that was "temporary" in 2024.

~95%of cloud security failures trace to human error / misconfig

~70%of cloud attack vectors lead with identity or credential compromise

277daverage time to detect a cloud breach

The number that should keep you up is the last one. Most of a calendar year, attacker resident in your cloud, undetected. That is not a detection-tooling problem. That is a visibility-and-ownership problem dressed up as one.

And misconfig never stays a single mistake. It chains. A leaked CI/CD token lands on an over-privileged role. That role reads a public store. The store holds another credential. That credential modifies production. Every link in that chain is a config nobody reviewed, because reviewing configs is boring and shipping features is not. Roughly 45% of breaches now touch the cloud, and the average public-cloud incident clears five million dollars. Boring is expensive.

Road two: the key someone sold

You do not need an exploit if you can log in. In 2026 that is the default. The clearest signal in the whole threat landscape: the majority of intrusions involve no malware at the entry point at all.

79%of intrusions are malware-free at initial access (stolen creds, not exploits)

54%of ransomware victims appeared in infostealer logs BEFORE the attack

~30%of incidents start with identity abuse, the top single vector

This is the Initial Access Broker economy, and it has matured into a clean professional supply chain. Infostealer malware harvests credentials and session cookies from some unlucky endpoint. Those logs get parsed, sorted, and sold. A broker packages "VPN access to a UK financial services firm, domain admin adjacent" and lists it. A ransomware affiliate buys it for four figures and is inside before your SOC has finished its morning standup.

The brokers have gone upmarket. Through 2025 and into 2026, researchers tracked IABs shifting toward high-value targets and premium pricing, with elite players retreating from burned public boards like the rebooted BreachForums toward closed, Russian-language platforms after names like IntelBroker got arrested and unmasked. The market did not collapse under law-enforcement pressure. It professionalised and went quieter.

The operational takeaway is brutal and simple. By the time ransomware detonates, the breach is old news. The encryption is the invoice, not the intrusion. Someone bought your access weeks earlier, sat inside, staged exfiltration, and only then pulled the trigger.

Road three: the code you imported

The third road is the one we walk down ourselves, voluntarily, thousands of times a day. Your dependency graph is a trust network you do not monitor and mostly cannot see. npm install is an instruction to download and execute arbitrary code from strangers, and 2026 has been a clinic in what that costs.

The shift started with Shai-Hulud in late 2025: a self-replicating worm that stole npm tokens and GitHub PATs, then used them to automatically republish itself into other packages. That ended the nuisance era. Then the consequences arrived at scale:

• Axios (March 2026): the most popular HTTP client in the ecosystem, 100M+ weekly downloads. A hijacked maintainer account pushed poisoned versions carrying a phantom dependency that dropped a cross-platform RAT on install. Attributed to a North Korean state actor. Live within 39 minutes of a stolen token being used.
• node-ipc (May 2026): three malicious versions published across multiple semver lines at once, deliberately maximising blast radius so anyone pinned to ^9, ^12, or a tilde range pulled an 80KB credential stealer on their next lockfile refresh.
• Miasma / @redhat-cloud-services (June 2026): a Shai-Hulud derivative abusing install-time scripts, compromising a vendor namespace through a hijacked GitHub account.

Notice the pattern. Almost none of these were typosquats you could have avoided by spelling carefully. They were legitimate, trusted, widely-used packages whose publishing identity got stolen. The malicious code arrived through the exact channel you told your build system to trust implicitly. postinstall is remote code execution that you opted into and put in your pipeline.

The throughline

Three roads, one root cause: trust granted without verification. The open cloud role, the stolen credential, the hijacked maintainer token. Same currency, different counter. We spent a decade buying perimeter products while the perimeter quietly redefined itself as "anyone holding a valid token." Identity is the perimeter now. It has been for a while. The breach reports are just catching the rest of us up.

// Opinion: this is good news, if you can stand to hear it

Every one of these is a hygiene failure, not a sophistication failure. Nobody out-galaxy-brained your defences. They used a checkbox, a credential, and an install script. That should be encouraging, because boring problems have boring fixes, and boring fixes are cheap relative to the next platform your vendor wants to sell you.

What actually moves the needle, in priority order:

• Least privilege, enforced, not aspirational. The CI/CD token that can reach prod is the entire kill chain compressed into one line. Scope it down until it hurts.
• Pin and verify dependencies. Lockfiles with integrity hashes. No blind caret ranges on anything that runs install scripts. Disable postinstall by default and allowlist the few that need it.
• Monitor infostealer logs for your own domains. If your credentials are for sale, you want to know before the affiliate does.
• Phishing-resistant MFA everywhere. When 79% of intrusions are malware-free, credentials are the product. Stop shipping the product.
• Default-deny cloud, reviewed IaC, CSPM that blocks instead of alerting into a void nobody reads.

None of this is a new product. It is discipline applied to the three roads attackers actually use. The movie hack is a distraction. The real breach is administrative, and so is the defence.

SOURCES // SentinelOne and StationX cloud + breach statistics (2026) // CrowdStrike 2025 (malware-free intrusions) // Verizon DBIR 2025 (infostealer logs, identity vectors) // Rapid7 threat intelligence, Initial Access Broker pricing shift (2026) // IBM X-Force Threat Intelligence Index 2026 // StepSecurity (node-ipc), Trend Micro and Microsoft (Axios / Sapphire Sleet), Wiz and Palo Alto Unit 42 (Miasma / Shai-Hulud). Figures are cross-source and approximate; treat as direction, not decimals.

#AppSec #CloudSecurity #SupplyChain #ThreatIntel #IAB #npm #DevSecOps

Anthropic, cannot give you anymore access to Mython and Fable, unless you are American military personnel....

root@elusive:~/posts$ cat no-universal-railguard.md

There Is No Universal Railguard, And They Shipped It Anyway

Filed under: agentic AI security // governance // things that were always going to happen

Anthropic told us the truth and we did not listen. Buried in the Fable 5 launch was one of the most honest sentences a frontier lab has ever published about its own safeguards: perfect jailbreak resistance is not currently possible for any model provider. Read that again. Not "we have not finished hardening." Not "edge cases remain." A flat statement that the unbreakable wall does not exist and will not exist on this architecture.

Then they put the model in front of hundreds of millions of people. Then a researcher beat the layer in under two days. Then the US government pulled the plug. None of these three events contradict the others. That is the whole point, and almost nobody is saying it.

The architecture, because the architecture is the story

Fable 5 and Mythos 5 are the same model. The difference is a classifier layer. When a query trips one of the high-risk buckets (cybersecurity, biology, chemistry, model distillation), Fable does not refuse. It silently downgrades the request to the weaker Opus 4.8 and tells you it did so. Mythos is the same model with the cyber classifiers lifted, handed to a small set of trusted defenders.

If you have ever deployed a WAF in front of an application you already understand the entire security posture here. The classifier is not the model's security. It is a request inspector bolted to the front. It reads what you send, scores it, and decides whether the real engine answers or the understudy does. It does not, and cannot, read your intent.

That is why the published bypass techniques are unremarkable to anyone in this field. Unicode and homoglyph substitution to dodge keyword matching. Long-context framing to dilute intent across a conversation so no single turn looks bad. Decomposition-recomposition, where you split a forbidden task into a dozen individually innocent sub-requests and reassemble the answer yourself. These are not exotic. They are the LLM equivalent of encoding a payload to slip past a signature-based filter. WAF evasion, new substrate.

So when the classifier layer falls, the correct reaction is not shock. The correct reaction is "yes, that is what classifier layers do." Anthropic said so themselves. Out loud. In the launch post.

Reading one: this is bad, and the takedown is the system working

Here is the uncomfortable version.

Anthropic has previously described Mythos-class capability as analogous to a cyberweapon that warrants careful oversight. Fine. Then the same company wrapped that capability in a layer it publicly admitted was defeatable in principle, tuned the layer conservatively, and shipped it to the general public at ten dollars per million input tokens. The safety argument rests entirely on three words: "no universal jailbreak." And the operative word in that phrase is yet.

A non-universal jailbreak is a key that opens one door and has to be re-cut for the next. A universal jailbreak is a master key. Anthropic's bet is that they can keep attackers stuck cutting individual keys, log every attempt, and patch faster than anyone can scale an attack. That is a reasonable bet for a monitored, narrow deployment to vetted defenders. It is a far shakier bet for a public model with hundreds of millions of users and a financial incentive sitting on every successful bypass.

In this reading, a government that recalls the model the moment a credible bypass surfaces is not overreacting. It is enforcing the precautionary principle the lab itself claimed to believe in. If your security control has a known expiry date and you sell it as if it does not, the recall is the smoke alarm doing its job. The fact that it is loud does not make it wrong.

Reading two: this is over-amplified, and partly a control play

Now the other version, which is also supported by the facts.

What did the disclosed bypass actually produce? By Anthropic's own account, the government's evidence was verbal, and the technique essentially amounts to asking the model to read a codebase and fix its flaws. That is not a weapon. That is Tuesday for every defender alive. The lurid screenshots, stack overflow exploit code and a meth synthesis pathway, describe capabilities you can already pull from other public frontier models and from a patient afternoon with a search engine. The leaked 120,000 character system prompt is not a compromise. It is the model's refusal logic and house style. It embarrasses, it does not hand over control, and system prompts get extracted from every frontier model by anyone who tries hard enough.

Then look at the plumbing of the takedown. Reporting points to the bypass being found by Amazon, which happens to be Anthropic's largest investor, a board presence, and its cloud host, then escalated to Treasury, then converted into a Commerce directive that pulled a model overnight. The White House framing is that Amodei was offered a fix-or-pull choice and refused. Anthropic's account differs on essentially every material point and says the letter arrived at 5:21pm with no technical specifics at all.

Strip the national-security wrapper and what is left is this: a model deployed to millions got recalled over a narrow, non-universal, verbally-described filter evasion, through a channel that runs straight through a competitor-and-investor. Apply that standard evenly and you do not have a safer industry. You have no new model releases at all, because every model in existence is vulnerable to non-universal jailbreaks by definition. That is not safety policy. That is a kill switch with a flag painted on it.

The AppSec verdict

Both readings are correct. That is the part that should keep you up at night, not either one alone.

The engineering claim is true. There is no universal railguard. Anybody selling you one is selling you a WAF and calling it a vault.

The product claim is where it breaks. "No universal bypass exists yet" is a dependency note, not a safety guarantee, and shipping it to the entire planet as if it were the latter is the actual unsafe act. Not the jailbreak. The framing.

The governance claim is the one that matters most to anyone who builds. A frontier model vanished for every customer, overnight, on the strength of a verbal, undocumented finding routed through an interested party. If your production workflow is coupled to a single closed API, you just watched a live demonstration of your own supply-chain risk. The model did not fail. The endpoint did not get hacked. It simply stopped existing because of a letter you will never read.

So treat "no universal jailbreak" as exactly what it is: the most honest thing the vendor said, and the one you are least allowed to forget. Build for the day the layer falls, because the people who built it already told you it would. Monitor like the control is temporary, because it is. And never put a production dependency somewhere a single letter can switch off at 5:21 on a Friday.

The railguard was never universal. The only surprise is that anyone is surprised.

// EOF  //  Elusive Thoughts  //  securityhorror.blogspot.com