The AI pentest that survives day three
Everyone is shipping autonomous pentesters. Almost nobody is shipping one that survives a real engagement. This is the backbone that does: Temporal underneath, Kali and Burp doing the work, a human holding the trigger.
August 2026 · offensive AI · durable execution · ~18 min read
I have an agentic pentest harness. It is a Kali box exposing 26 tools over SSH through an MCP server, an LLM planning the next move, and enough glue to let it run recon, enumeration and a first pass at exploitation without me touching the keyboard. On a lab target it is genuinely impressive. It chains subfinder into httpx into nmap, reads the output, picks a plausible next step, and keeps going.
Then I pointed it at a real engagement. It died on day three.
Not because the model was stupid. Because the LLM API rate-limited me halfway through a scan and the loop threw an exception and unwound. Because a forty minute nmap run against a large range hung and I had no way to tell whether it was alive or wedged. Because the Kali VPS rebooted overnight and took every scrap of in-memory state with it. Because at one point the agent stopped running commands and started imagining their output, confidently reporting findings for hosts it had never touched. That last one has a name now. The EnIGMA researchers called it soliloquising, and once you have watched an agent do it you stop trusting anything it says without a receipt.
The intelligence was fine. The execution layer under it was cardboard. My harness was a clever brain bolted onto a fragile nervous system, and a real engagement is exactly the environment that finds every fragile thing you shipped.
This post is about the nervous system. It is about wrapping the whole workflow in durable execution with Temporal, so the engagement retries when the model flakes, resumes when the box dies, waits days for a human to approve an exploit, and writes an audit trail you could hand to a client or a regulator without flinching. Kali and Burp still do the actual work. Temporal is what makes the work trustworthy enough to put your name on.
Code is Python throughout, because that is what the harness is already written in. Let me start with why the obvious build is a trap.
The build everyone reaches for first
Strip away the framework of the week and every agentic pentester is the same loop. Ask the model what to do next. Run it. Feed the result back. Repeat until it thinks it is finished.
# the harness, roughly. runs great in a demo.
context = []
while not done:
step = llm.plan(context) # what should we do next?
result = ssh_kali(step.command) # run it on the Kali box
context.append((step, result)) # remember it... in RAM
done = llm.is_finished(context)
Five lines. Every failure from the intro is already in here.
context is a list in memory, so when the VPS reboots the engagement is gone. ssh_kali raises on a dropped connection and the whole loop unwinds. llm.plan returns a 429 under load and the loop unwinds again. There is no point in those five lines where a human approves anything before it fires, and there is no record of what happened beyond whatever you thought to print at the time.
So you start bolting things on. A try/except around the SSH call. A retry decorator on the model call. Postgres for the context so it survives a reboot. An approvals table with a poller. A logging pipeline so you have some idea what ran. Six months later you have written a worse, undocumented, untested version of a durable execution engine, and you are maintaining it alone.
Durable execution is the part of the agent nobody demos and everybody eventually rebuilds badly.
Temporal is that engine, done properly. Before the architecture, here is the whole argument in one table.
| When this happens | Agent as a script | Durable workflow |
|---|
| The box crashes mid-engagement | state lost, start from zero | resumes at the exact step |
| LLM returns 429 or 5xx | exception unwinds the run | activity retries with backoff |
| A 40-minute nmap run | hung or working? no idea | heartbeat proves liveness |
| Exploit needs human sign-off | an input() prompt, or nothing | signal, waits for days |
| Client asks what you ran | whatever you logged | full event history |
| Target drifts out of scope | a hope and a comment | non-retryable guard refuses |
| 200 hosts to cover | gather() and pray | child workflow per host, bounded |
Same model, same tools. The right-hand column is what a backbone buys you.
The architecture
Here is the whole thing on one page. Read it top to bottom. A trigger starts a workflow, the workflow orchestrates activities, the activities drive Kali and Burp through a governance layer, and everything that happens lands in an event history that doubles as your evidence and your metrics.
Figure 1 / end-to-end architecture
Orchestration · Temporal
Trigger
code change · CI · schedule
→
PentestWorkflow
holds state · zero I/O
↔
🔒 Human gate
approval signal
↓
Activities · all I/O lives here
↓
Governance · every tool call
Tool gateway
identity · scope · risk → allow / block / redact / escalate
hardened hooks
last guard before action
↓
Execution
Kali box
MCP harness · 26 tools · SSH · nmap, nuclei, sqlmap, amass
Burp Suite DAST
GraphQL · create_schedule_item
↓
Scoped targets · rules of engagement enforced in code
↓
Durability · the payoff
Event history
every call + return, replayable
→
Audit trail
chain of custody
→
SDLC KPIs
coverage · MTTR · FP rate
Figure 1. The workflow never touches the network. Everything with a blast radius is an activity, behind governance, against a scoped target, recorded in history.
The single most important rule is the boring one at the top. The workflow orchestrates and holds state. It never does I/O. Every call to the model, every tool run, every Burp scan, every line written to a report is an activity. That split is not a style preference. It is the thing that makes the workflow replayable, and replay is where all the durability comes from. I will come back to it.
Notice the band in the middle. The agent driving this pentest is still an agent, and it gets governed like every other agent in the estate. Tool calls route through a gateway that checks four things in order: which agent this is, whether it is allowed to reach this tool at all, what it may do inside it, and whether this specific call is safe. Its access is just-in-time, granted for the engagement and pulled afterwards, so a compromised pentest agent can reach whatever it holds in that window and nothing else. Offensive tooling is the last thing in your estate you want holding standing production credentials.
Scope follows the diff
The trigger box in Figure 1 says code change on purpose. The useful version of this system does not wait for a quarterly booking. It starts when a pull request does, and the scope of the test is derived from what actually changed.
A diff that adds one endpoint does not need an estate sweep. It needs that endpoint hammered. A change across the API surface needs the whole web app API scanned. A Terraform change needs the cloud infra looked at, not the login form. Same engine, different scope, decided automatically from the diff.
Figure 2 / scope follows the diff
→
diff classifier
derive scope + task queue
↓
targeted
one new endpoint → Burp API scan + authz/BOLA probes
api-wide
whole web app API → DAST over the OpenAPI spec + nuclei
infra
cloud / Terraform → recon + misconfig & exposure sweep
full
nightly / weekly → child workflow per host
Figure 2. One dispatcher, four scopes. The blast radius of the test matches the blast radius of the change.
| What changed | Derived scope | What the workflow runs |
|---|
One new route, e.g. POST /v2/payouts | that route + its auth | Burp API scan of the route, BOLA / authz probes, an llm_plan focused on the new surface |
| Broad API change, OpenAPI spec bumped | whole web app API | Burp DAST driven from the spec, nuclei templates, parameter fuzzing |
| Terraform / IaC change | cloud infra in the blast radius | recon, exposed-service sweep, nuclei misconfig checks against the changed footprint |
| Nightly schedule | full agreed scope | everything, fanned out one child workflow per host |
The dispatcher is thin. A CI step or a webhook inspects the changed paths, builds a scope, and starts the right workflow. The important detail is the workflow ID.
from temporalio.client import Client
from dataclasses import dataclass
@dataclass
class ScopeSpec:
kind: str # "targeted" | "api_wide" | "infra" | "full"
targets: list[str] # hosts, base URLs, or cloud accounts in scope
openapi: str | None = None
engagement_id: str = ""
def classify(changed: list[str]) -> ScopeSpec:
if any(p.endswith(".tf") or p.startswith("infra/") for p in changed):
return ScopeSpec("infra", targets=cloud_scope())
if "openapi.yaml" in changed:
return ScopeSpec("api_wide", targets=[API_BASE], openapi="openapi.yaml")
routes = [p for p in changed if p.startswith("api/routes/")]
if routes:
return ScopeSpec("targeted", targets=routes_to_urls(routes))
return ScopeSpec("full", targets=full_scope())
async def dispatch(pr: str, sha: str, changed: list[str]) -> None:
client = await Client.connect("temporal:7233", namespace="pentest")
scope = classify(changed)
scope.engagement_id = f"{pr}-{sha[:12]}"
await client.start_workflow(
PentestWorkflow.run,
scope,
id=f"pentest-{scope.engagement_id}", # one run per PR+sha, re-triggers dedupe
task_queue=f"pentest-{scope.kind}", # infra work runs off its own worker pool
)
The workflow ID is keyed to the PR and commit, so pushing the same commit twice does not start two engagements. The task queue is keyed to the scope kind, so a cloud scan never lands on the workers that run web tests.
Because it starts in CI, it gates in CI. The workflow returns a verdict, the PR check goes red on a confirmed high, and every run stamps the same event history you will later mine for coverage-per-change and time-to-first-finding. This sits right beside the SAST and SCA already running on the diff. Same trigger, deeper test.
Temporal in five minutes, for people who break things
Three concepts and one rule.
Workflows and activities
A workflow is your orchestration logic. It decides what happens next. An activity is anything that touches the outside world: a model call, a shell command on Kali, a Burp scan, a write to disk. Activities are allowed to be slow, flaky and failure-prone, and Temporal wraps every one of them in configurable timeouts and retries.
The rule from Figure 1, restated because it is the whole game: orchestration goes in the workflow, I/O goes in activities. Break that and everything below stops working.
Determinism and replay
Temporal does not snapshot your process memory. It records an event history: every activity the workflow scheduled and every result that came back. When a worker dies, a fresh one replays that history through your workflow code, rebuilds the exact in-memory state, and carries on from the last completed step. Nothing re-runs that already finished.
That is why the workflow has to be deterministic. A raw datetime.now(), a bare random(), a direct HTTP call, any of them make the replay diverge from the recorded history and Temporal will refuse it. You get deterministic replacements for the safe things, workflow.now() and workflow.random(), and activities for everything else. It feels like a constraint for about a day, then it feels like the reason the thing never loses state.
The one thing to rememberThe event history that makes the workflow crash-proof is the same artefact that makes it auditable. You are not building durability and evidence separately. They are the same log.
The engagement as a workflow
Model the engagement as a state machine. Each phase is durable, so the workflow always knows where it is, even after the third worker restart of the week.
Figure 3 / engagement state machine
↓
Figure 3. Every box is a persisted state. The amber one does not advance until a human sends a signal, and it will wait for days without holding anything open.
In code the skeleton is small. State lives in plain instance attributes, which is exactly what Temporal rebuilds on replay. A query exposes progress to a dashboard without interrupting the run.
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
with workflow.unsafe.imports_passed_through():
from .activities import llm_plan, run_tool, burp_scan, exploit, write_report
from .types import ScopeSpec, Finding, Approval
@workflow.defn
class PentestWorkflow:
def __init__(self) -> None:
self.phase: str = "scoping"
self.findings: list[Finding] = []
self.approval: Approval | None = None
@workflow.query
def status(self) -> dict:
return {"phase": self.phase, "findings": len(self.findings)}
@workflow.run
async def run(self, scope: ScopeSpec) -> dict:
self.phase = "recon"
recon = await workflow.execute_activity(
run_tool, ("recon", scope),
start_to_close_timeout=timedelta(minutes=45),
heartbeat_timeout=timedelta(seconds=60),
retry_policy=RetryPolicy(maximum_attempts=4),
)
self.phase = "analysis"
self.findings = await self._analyse(scope, recon) # llm_plan + burp_scan, below
approved = await self._await_gate() # the human gate, below
if approved:
self.phase = "exploit"
await self._exploit_confirmed(scope)
self.phase = "report"
return await workflow.execute_activity(
write_report, (scope, self.findings),
start_to_close_timeout=timedelta(minutes=5),
)
The run method reads like a plan of the engagement because that is what it is. The interesting parts, the analysis, the gate and the exploit, are next.
Driving Kali and Burp as activities
This is where the real work lives, and it is deliberately dull. An activity is a normal async function with a decorator. It can be as slow and unreliable as the tool it wraps, because the workflow above it does not care how many times it had to retry, only that it eventually returned.
Kali, over the harness
The Kali side goes through the MCP harness: 26 tools exposed over SSH. A single recon activity picks the right tool, checks the target is in scope, and streams the output back while heartbeating so Temporal can tell a live scan from a wedged one.
from temporalio import activity
from .mcp import kali # MCP client to the Kali harness (SSH)
@activity.defn
async def run_tool(kind: str, scope: ScopeSpec) -> ToolResult:
tool = pick_tool(kind, scope) # nmap, nuclei, amass, ...
guard_scope(tool.target, scope) # refuse out-of-scope (see below)
async with kali.stream(tool) as proc: # a 40-minute nmap is fine here
async for line in proc.stdout:
activity.heartbeat(proc.progress) # "I am alive, here is where I am"
if activity.is_cancelled(): # timed out or workflow cancelled
await proc.terminate()
raise
return ToolResult(tool=tool.name, raw=proc.output, target=tool.target)
The heartbeat is the whole point. If Kali stops sending progress for longer than the heartbeat_timeout set at the call site, Temporal declares the activity dead and reschedules it, without waiting out the full 45-minute start-to-close window.
Burp, over the DAST GraphQL API
Burp Suite DAST exposes a GraphQL API. You launch a scan with the create_schedule_item mutation and poll the scan query for status and issues. Two operations:
# the two Burp operations
CREATE_SCAN = """
mutation($site_ids: [ID!]!) {
create_schedule_item(input: {site_ids: $site_ids}) {
schedule_item { id scheduled_run_time }
}
}"""
SCAN_STATUS = """
query($id: ID!) {
scan(id: $id) {
status
issue_counts { total }
issues(start: 0, count: 1000) { serial_number type_index severity }
}
}"""
Wrapped in an activity, launch then poll, heartbeating on every tick:
import asyncio, httpx
from temporalio import activity
@activity.defn
async def burp_scan(base_url: str, openapi: str | None) -> list[Finding]:
async with httpx.AsyncClient(headers=burp_auth()) as http:
site = await ensure_site(http, base_url, openapi) # idempotent: reuse if exists
started = await gql(http, CREATE_SCAN, {"site_ids": [site]})
scan_id = started["create_schedule_item"]["schedule_item"]["id"]
while True:
s = await gql(http, SCAN_STATUS, {"id": scan_id})
status = s["scan"]["status"]
activity.heartbeat(status)
if status in ("succeeded", "failed"):
break
await asyncio.sleep(15)
return [to_finding(i) for i in s["scan"]["issues"]]
One detail that will bite you: activities retry, so launching a scan must be idempotent. ensure_site reuses a site if it already exists, and you tag the scan with the engagement ID so a retry after a network blip rejoins the existing scan instead of starting a second one. More on idempotency where it actually hurts, further down.
The planner is an activity, and that is the whole trick
The model is the least reliable component in the system. It rate-limits, it times out, it occasionally refuses on content policy, and once in a while it hands back confident nonsense. So it goes in an activity, sealed off from the workflow, wrapped in a retry policy, and backstopped by a fallback model. The workflow never sees the flakiness. It asks for a plan and eventually gets one, or a clean, loud failure.
Figure 4 / the planner never takes the engagement down
llm_plan
activity · non-determinism sealed inside
↓
✓ ok
plan validated against scope, returned to workflow
↻ 429 / 5xx / timeout
retry with backoff, then fall back to another model
✗ auth / refusal / malformed
non-retryable, fail fast, surface it
Figure 4. Transient failures are Temporal's job. Hard failures fail loudly rather than silently corrupting the run. Nothing here reaches the workflow as a half-state.
The retry policy is set where the activity is called, not on the activity itself. Transient errors retry with exponential backoff. A short list of error types is marked non-retryable, because retrying an auth failure or a malformed plan six times just wastes ninety seconds before failing anyway.
from temporalio.exceptions import ActivityError
@activity.defn
async def llm_plan(ctx: PlanContext, model: str) -> Plan:
resp = await llm(model).plan(ctx) # RateLimited / AuthError / PolicyRefusal
return validate_plan(resp) # malformed → MalformedPlan (non-retryable)
# ...called from the workflow, with the policy and a one-shot fallback:
async def _next_step(self, ctx: PlanContext) -> Plan:
policy = RetryPolicy(
initial_interval=timedelta(seconds=2),
backoff_coefficient=2.0,
maximum_attempts=6,
non_retryable_error_types=["AuthError", "PolicyRefusal", "MalformedPlan"],
)
try:
return await workflow.execute_activity(
llm_plan, (ctx, PRIMARY_MODEL),
start_to_close_timeout=timedelta(seconds=90), retry_policy=policy)
except ActivityError: # primary exhausted or hard-failed
return await workflow.execute_activity(
llm_plan, (ctx, FALLBACK_MODEL),
start_to_close_timeout=timedelta(seconds=90), retry_policy=policy)
Six attempts with backoff rides out a rate-limit storm. A hard model outage falls through to a second provider once. Only then does the phase fail, and it fails with a full history of every attempt.
There is a subtler win here, and it is the one that matters for trust. The model is non-deterministic, but that non-determinism is trapped inside the activity. All the workflow records in its history is llm_plan returned this plan. Replay the engagement a year later and it rebuilds identically, even though the model that produced the plan would answer differently today. The audit trail is stable because the chaos is quarantined.
The model proposes. The workflow disposes. A plan is still checked against the rules of engagement before a single packet leaves the box.
The human gate
Recon and scanning can run unattended all day. Exploitation cannot. Before anything with a blast radius fires against a client's production, a human approves it. In most agent frameworks that is a fragile bit of glue: a blocking prompt, or a database row and a poller someone wrote on a Friday. In Temporal it is a first-class primitive. The workflow blocks on a condition, a signal releases it, and a durable timer covers the case where nobody answers.
Figure 5 / approval before anything fires
Analysis
proposed exploit + evidence
→
🔒 GATE
wait_condition(approved) · 24h durable timer
↓
signal: approve
→ run the exploit
signal: reject
→ skip straight to report
timer fires, no answer
→ escalate, do not exploit
Figure 5. The workflow can sit at this gate over a weekend. It holds no thread, no connection, no process. The timer and the wait survive every restart in between.
import asyncio
@workflow.signal
def approve(self, decision: Approval) -> None:
self.approval = decision # who, when, approved bool, note
async def _await_gate(self) -> bool:
self.phase = "awaiting_approval" # now visible to the dashboard via status()
try:
await workflow.wait_condition(
lambda: self.approval is not None,
timeout=timedelta(hours=24),
)
except asyncio.TimeoutError:
await workflow.execute_activity(
escalate, self.findings,
start_to_close_timeout=timedelta(minutes=1))
return False # nobody answered in 24h: refuse to exploit
return self.approval.approved
The signal can arrive from Slack, a web UI, a phone. Wherever it comes from, the approver, the timestamp and the note land in the event history. Who authorised the exploit becomes part of the evidence, not a memory.
This is also the answer to a governance question that keeps security leaders up at night. There is no override on this gate, because an override that exists gets used under deadline pressure at exactly the wrong moment. The gate is part of the machine, not a step in a runbook that a tired human can wave through. The automation physically cannot proceed to exploitation without a recorded human decision.
This is where the pentester livesThe gate is not scaffolding you remove once the model gets good enough. It is the point of contact between machine breadth and human judgment. Deciding whether an exploit is safe to fire against production, whether the finding is real, whether the chain is worth pursuing, that is the job. The workflow just makes the decision unskippable and permanent.
Rules of engagement, enforced in code
An out-of-scope target is not a retryable error. It is a full stop. The model can hallucinate a target all it likes. It hits a wall written in deterministic Python, not a polite request in a system prompt.
from temporalio.exceptions import ApplicationError
def guard_scope(target: str, scope: ScopeSpec) -> None:
if not in_scope(target, scope):
raise ApplicationError(
f"target {target} outside {scope.kind} scope",
type="OutOfScopeError",
non_retryable=True, # do not retry a scope violation
)
Called at the top of every tool and exploit activity. A refusal is recorded in history with the target and the reason, which is itself proof you stayed inside the lines.
That guard is the innermost ring. Outside it, the gateway from Figure 1 blocks any tool call the agent is not authorised to make, and the just-in-time credentials never reach out-of-scope systems in the first place. If the workflow logic were somehow wrong, two more layers still hold, and each one records what it refused. Scope is not a promise in a statement of work. It is enforced three times and logged every time.
The event history is your evidence
Here is the part that turns an AI-assisted test into something you can defend. The event history Temporal keeps so it can survive a crash is, without any extra work, the most complete record of an engagement you have ever had. It looks like this.
# excerpt, PentestWorkflow event history
09:14:02 WorkflowStarted scope=api_wide pr=418 sha=4b1c9e0
09:14:03 ActivityScheduled run_tool(recon)
09:41:20 ActivityCompleted run_tool(recon) 214 hosts, 9 services
09:41:21 ActivityScheduled burp_scan(base=api.acme.test)
10:58:07 ActivityCompleted burp_scan 17 issues
10:58:08 ActivityScheduled llm_plan(PRIMARY_MODEL)
10:58:10 ActivityFailed llm_plan RateLimited (attempt 1/6)
10:58:14 ActivityCompleted llm_plan plan: probe BOLA on /v2/payouts
10:58:15 TimerStarted gate-timeout 24h
11:03:44 SignalReceived approve by=alex note="prod ok, low blast radius"
11:03:45 ActivityScheduled exploit(finding=BOLA-/v2/payouts)
11:07:12 ActivityCompleted exploit confirmed, evidence stored
Every decision, every retry, every approval, timestamped and immutable. Note the rate-limit at 10:58:10 that retried and succeeded four seconds later. The engagement never noticed.
Figure 6 / one log, two products
Event history
every scheduled + completed event
→
Evidence / chain of custody
what ran, when, against what, who approved · replay it a year later
SDLC KPIs
coverage · time-to-first-finding · MTTR · false-positive rate · % needing a human · analyst-hours saved
Figure 6. Because the metrics are computed from the log rather than self-reported, time-to-first-finding is literally the first finding event minus the start event. Nobody is massaging a spreadsheet.
Your numbers go hereThe KPIs are emitted per run, but the baselines are yours. Drop in your real coverage percentage, your median time-to-first-finding, your false-positive rate before and after the gate, and the analyst-hours the automation gave back last quarter. Those are the numbers that turn this from an architecture into a business case.
Scaling without nuking the client
The nightly full-scope run and the api-wide scan both fan out across many targets. The wrong way is asyncio.gather over every host at once, which is a great way to knock over the thing you were hired to test. The right way is a child workflow per host, launched in bounded batches.
@workflow.defn
class FullScopeWorkflow:
@workflow.run
async def run(self, scope: ScopeSpec) -> list[dict]:
results = []
for batch in chunked(scope.targets, size=10): # never more than 10 hosts at once
results += await asyncio.gather(*[
workflow.execute_child_workflow(
PentestWorkflow.run,
host_scope(scope, host),
id=f"pentest-{scope.engagement_id}-{host}",
task_queue="pentest-host",
)
for host in batch
])
return results
Each host is its own child workflow, with its own history, its own retries and its own gate. One host falling over does not take the campaign with it, and the parent history stays small.
The real throttle is the task queue. Workers pull work from pentest-host, so sizing that worker pool caps how many host scans run concurrently across every engagement at once, not just within one. And for a long-lived campaign that watches a scope continuously, the history would grow without bound, so you call continue_as_new to start a fresh history while carrying the state forward.
if workflow.info().get_current_history_length() > 10_000:
workflow.continue_as_new(carry_forward(self.state)) # fresh history, same campaign
The honest costs
This is not free, and pretending it is would be the kind of vendor pitch this blog exists to mock.
The determinism tax is real. You cannot reach for datetime.now() or a stray HTTP call inside the workflow. Everything with a side effect becomes an activity. It is a genuine discipline and it will trip you up in the first week, until it becomes the reason nothing ever loses state.
Idempotency is the sharp edge. Temporal executes activities at least once, which means an activity can run twice, for instance after a heartbeat timeout on a scan that was merely slow rather than dead. A recon scan running twice is wasteful. A destructive exploit running twice is an incident. So exploit activities carry an idempotency key per finding, and the genuinely dangerous ones sit behind the human gate, where a double-fire would need a double-approval that is never going to happen by accident. Design this in from the start. It is the one place the model of at-least-once bites hard.
You now run infrastructure. A Temporal cluster or Temporal Cloud, workers, task queues, a namespace. For a solo consultant chasing one engagement that is heavier than a Python script and probably not worth it. For a team running continuous, change-triggered testing across an estate, it pays for itself the first time an engagement survives a 3am worker reboot that would otherwise have cost a day.
The harness is still yours. Temporal makes the MCP tools, the Burp integration and the plan validation reliable. It does not make them exist. That work is still on you, and the plan-validation layer in particular needs care, because it is the thing standing between a hallucinated plan and a real packet.
Why the pentester is still the point
Step back from the plumbing and look at what the machine is actually good at, because the honest version of this argument needs both halves.
The autonomous tools are strong at the front of the pipeline. Given a one-day CVE with the advisory in hand, a current model exploits it around 87% of the time. Take the description away and that falls to 7%, which tells you precisely what kind of strong it is: fast at applying what is already written down. Recon chaining is reliable. SSRF and injection are the highest-performing classes across every benchmark. None of it gets bored on host two hundred.
Now the other half. Point the same class of agent at real vulnerabilities instead of sanitised lab ones and success drops to about 13%, and to nearly zero on hard targets. Roughly 70% of critical web vulnerabilities are business logic flaws, and no autonomous agent detects those reliably, because a business logic flaw only exists in the context of how your business is meant to work. Chained exploitation across five or more conditional steps is still out of reach. The first head-to-head test on a live enterprise network, 8,000 hosts, had the AI find 9 valid issues while the best human found 13, at around 18 dollars an hour of value against 60 for the professional.
The headline case proves the rule. XBOW topped HackerOne's US leaderboard with over a thousand submissions in a few months. Read past the headline: 132 were confirmed and resolved, hundreds were duplicates or informative, and every finding was reviewed by human staff before submission. The autonomous tool that beat the humans still ran on humans.
So the shape of the win is not subtle. The AI does the volume. The human owns the call. Durable execution is what lets you run the two together across a real engagement, ride out the model's bad days, hold the exploit until a person says go, and hand over a log you can replay line by line. The gate in Figure 5 is not the automation's weakness. It is the automation being honest about where judgment belongs.
Build the backbone first
One thing to take away: a model without durable execution is a demo, and a demo is not an engagement. Build the nervous system before you fall in love with the brain. Wrap the workflow in Temporal, put every model call and every tool call in an activity, gate the dangerous phase behind a human signal, and let the event history be both your crash recovery and your evidence.
Do that and the AI stops being a party trick and becomes a force multiplier you can bill for. The pentester goes back to doing the part that was always the actual job. And the client gets a report backed by a trail you can replay a year later, which is the only kind of trust that survives contact with a real audit.
My harness started as the brain. Temporal is the nervous system that made it worth pointing at anything that matters.
