Building a Pentest AI Agent: First Field Test Results

Layer8 began R&D on a pentest AI agent in 2025/5 and conducted its first field test in 2026/1.

In this post, we introduce the agent’s architecture, design philosophy, and field test results.

TL;DR

Developed a fully autonomous pentest AI agent for web applications.
- Implemented the entire workflow — from initial reconnaissance to report generation — as a multi-agent system.
- Built a man-in-the-middle proxy (L7 Egress Gateway) for scope control, throttling, and inspection.
Field test results (as of 2026/2/8):
- HackerOne
  - Tested against 30 target URLs across VDP/BBP programs, achieving #86 worldwide in the VDP leaderboard (90 days).
  - VDP: 3 vulnerability reports triaged at the U.S. Department of Defense (DoD).
  - BBP: 3 submissions (duplicates).
- Pro bono PoC (1 company)
  - Discovered multiple exploitable vulnerabilities, including Stored XSS.
Benchmark
- PortSwigger Academy Lab: 84%

What This Is / What This Is Not

This is:

A fully autonomous AI agent that performs black-box/gray-box testing of web applications.
Accuracy-focused and scalable regardless of target size.

This is not:

A bot that automatically submits vulnerability reports.
A general-purpose internet crawler or attack toolkit.
A replacement for coverage-oriented vulnerability assessments (for now).
A white-box testing or network-level scanner.

Architecture

---
config:
  layout: dagre
  theme: base
---
flowchart LR
    %% Entry Point
    U(("User")) -- Target URL --> SA

    %% Control Plane
    subgraph Control_Plane["Control Plane"]
        SA["Strategist Agent"]
    end

    %% Core Components
    subgraph Blackboard["Blackboard"]
        direction LR
        TQ[("Task Queue")]
        FG[("Findings Graph")]
        HT[("Knowledge Base")]
    end

    %% Tactical Plane
    subgraph Tactical["Tactical Plane"]
        direction LR

        subgraph Agents["　"]
            direction LR
            R["Recon Agent"]
            V["Vulnerability Agent"]

            subgraph E_Unit["Exploit Unit"]
                direction TB
                E["Exploit Agent"]
                E --> SQLI["SQLi"]
                E --> XSS["XSS"]
                E --> ETC["..."]
            end
        end

        Agents ==> Toolset

        subgraph Toolset["Toolset"]
            direction TB
            BW["Web Browser"]
            SH["Kali Shell"]
            PY["Python"]
            PI["Proxy Inspector"]
        end
    end

    %% Validation Layer
    subgraph Validation["Validation Layer"]
        direction TB
        VA["Verification Agent"]
        RA["Report Agent"]
    end

    %% Infrastructure
    P_Svc{{"MITM Proxy"}}
    TWA(("Target Web App"))

    %% Logic Flow
    SA <--> Blackboard
    SA -. "Dispatch" .-> R & V & E & VA & RA
    SA -- "Push/Pop" --> TQ

    %% Output & Verification Flow
    Tactical -- Exploit PoC --> VA
    VA -- Confirmed --> RA
    Validation <--> Blackboard

    %% Traffic Flow
    Toolset -- Traffic --> P_Svc
    VA -- Verification Traffic --> P_Svc
    P_Svc -- Request --> TWA
    TWA -- Response --> P_Svc

%% Style Definitions
    style U fill:#f8f9fa, stroke:#212529, stroke-width:2px
    style SA fill:#eef6ff, stroke:#007bff, stroke-width:2px
    style Control_Plane fill:#ffffff, stroke:#007bff, stroke-width:2px, stroke-dasharray: 5 5

    style Blackboard fill:#f8f9ff, stroke:#5c6bc0, stroke-width:2px
    style TQ fill:#ffffff, stroke:#5c6bc0, stroke-width:1px
    style FG fill:#ffffff, stroke:#5c6bc0, stroke-width:1px
    style HT fill:#ffffff, stroke:#5c6bc0, stroke-width:1px

    style Tactical fill:#ffffff, stroke:#333333, stroke-width:2px, stroke-dasharray: 5 5
    style R fill:#f0f7ff, stroke:#2196f3, stroke-width:1px
    style V fill:#f3e5f5, stroke:#9c27b0, stroke-width:1px
    style E_Unit fill:#fff5f5, stroke:#f44336, stroke-width:1px, stroke-dasharray: 2 2
    style E fill:#ffebee, stroke:#f44336, stroke-width:1px
    style SQLI fill:#ffffff, stroke:#f44336, stroke-width:1px
    style XSS fill:#ffffff, stroke:#f44336, stroke-width:1px
    style ETC fill:#ffffff, stroke:#f44336, stroke-width:1px

    style Toolset fill:#fffaf0, stroke:#fb8c00, stroke-width:2px
    style P_Svc fill:#e8f5e9, stroke:#4caf50, stroke-width:2px
    style TWA fill:#f5f5f5, stroke:#757575, stroke-width:2px
    style Validation fill:#ffffff, stroke:#28a745, stroke-width:2px, stroke-dasharray: 5 5

Design Philosophy

Some proprietary details have been omitted.

Multi-Agent System:

We adopted a multi-agent system in which multiple specialized agents collaborate to execute a pentest.

Some notable OSS projects, such as westonbrown/Cyber-AutoAgent, are built as a single agent. There is no definitive answer as to which approach is better, but we chose a multi-agent system — accepting the added complexity of managing multiple components — for the following reasons:

XBOW, a leading project in this space, also adopted a multi-agent system.
Easier per-phase tuning:
- Models, system prompts, tools, and guardrails can be flexibly designed for each phase.
- Mechanical false-positive checkers can be implemented in an optimized form for each phase.
- Certain phases benefit from unique state management and forgetting mechanisms.

Findings Graph as SSOT (Single Source of Truth):

The Findings Graph serves as the SSOT for centrally managing intelligence throughout the entire attack chain. This mechanism provides the following benefits:

Ensuring context consistency across the entire attack chain.
Mitigating the risk of forgetting critical information.
Improving token efficiency through mechanical extraction of task-relevant subgraphs.

The Findings Graph is not merely a knowledge base — it is a critical foundation that plays a central role in orchestration. As the Findings Graph infrastructure encapsulates our core proprietary know-how, its details remain confidential.

System-Level Guardrails:

Since full automation without HITL (Human-in-the-Loop) is a core concept, guardrails that ensure safe operation without human oversight are essential.

In addition to prompt-level guardrails, a man-in-the-middle proxy (L7 Egress Gateway) functions as a system-level guardrail. This prevents unintended attacks and excessive requests, ensuring the safety of the fully autonomous pentest AI agent.

Field Test

HackerOne

We tested against a total of 30 target URLs under the following conditions:

Time limit: 1 hour
Input: URL only (black-box testing)

The results were as follows:

VDP (Vulnerability Disclosure Program)
- 3 vulnerability reports triaged at the U.S. Department of Defense (DoD).
- Achieved #86 worldwide in the 90-day leaderboard.
BBP (Bug Bounty Program)
- 3 submissions (duplicates).

Having confirmed the agent’s effectiveness against production systems, we have concluded our HackerOne experiments and are now focusing on developing proprietary benchmarks and tuning, as described below.

HackerOne Ranking

Pro Bono PoC

With the cooperation of a partner company, we conducted a fully autonomous pentest as a pro bono PoC using our in-development pentest AI agent. In a gray-box test with minimal information provided (URL and credentials), the agent successfully discovered multiple exploitable vulnerabilities, including Stored XSS.

Benchmark

PortSwigger Academy Lab

In experiments conducted with a 1-hour time limit, the agent solved 84% of the labs as of 2025/12/2.

PortSwigger Academy Lab Score

Proprietary Benchmark

To fully leverage the advantage of easier per-phase tuning, a benchmark for evaluating tuning changes is essential. While the XBOW Benchmark and PortSwigger Academy Lab are useful for evaluating the entire attack chain, we believe they are not well-suited for per-phase tuning.

To address this, we are developing a proprietary benchmark for per-phase performance improvement and cost optimization. Details will be shared in a future blog post.

Conclusion

We developed a fully autonomous pentest AI agent for web applications and achieved promising results in our first field test. We will continue R&D to automate more advanced attack chains and expand the range of supported targets.

We are looking for companies willing to participate in proof-of-concept testing of our pentest AI agent. If you are interested, please reach out to us.