Blog

08 Feb, 2026

Building a Pentest AI Agent: First Field Test Results

Layer8 began R&D on a pentest AI agent in 2025/5 and conducted its first field test in 2026/1. In this post, we introduce the agent's architecture, design philosophy, and field test results. TL;DRDeveloped a fully autonomous pentest AI agent for web applications. Implemented the entire workflow — from initial reconnaissance to report generation — as a multi-agent system. Built a man-in-the-middle proxy (L7 Egress Gateway) for scope control, throttling, and inspection.Field test results (as of 2026/2/8): HackerOne Tested against 30 target URLs across VDP/BBP programs, achieving #86 worldwide in the VDP leaderboard (90 days). VDP: 3 vulnerability reports triaged at the U.S. Department of Defense (DoD). BBP: 3 submissions (duplicates).Pro bono PoC (1 company) Discovered multiple exploitable vulnerabilities, including Stored XSS.Benchmark PortSwigger Academy Lab: 84%What This Is / What This Is Not This is:A fully autonomous AI agent that performs black-box/gray-box testing of web applications. Accuracy-focused and scalable regardless of target size.This is not:A bot that automatically submits vulnerability reports. A general-purpose internet crawler or attack toolkit. A replacement for coverage-oriented vulnerability assessments (for now). A white-box testing or network-level scanner.Architecture --- config: layout: dagre theme: base --- flowchart LR %% Entry Point U(("User")) -- Target URL --> SA %% Control Plane subgraph Control_Plane["Control Plane"] SA["Strategist Agent"] end %% Core Components subgraph Blackboard["Blackboard"] direction LR TQ[("Task Queue")] FG[("Findings Graph")] HT[("Knowledge Base")] end %% Tactical Plane subgraph Tactical["Tactical Plane"] direction LR subgraph Agents["　"] direction LR R["Recon Agent"] V["Vulnerability Agent"] subgraph E_Unit["Exploit Unit"] direction TB E["Exploit Agent"] E --> SQLI["SQLi"] E --> XSS["XSS"] E --> ETC["..."] end end Agents ==> Toolset subgraph Toolset["Toolset"] direction TB BW["Web Browser"] SH["Kali Shell"] PY["Python"] PI["Proxy Inspector"] end end %% Validation Layer subgraph Validation["Validation Layer"] direction TB VA["Verification Agent"] RA["Report Agent"] end %% Infrastructure P_Svc{{"MITM Proxy"}} TWA(("Target Web App")) %% Logic Flow SA <--> Blackboard SA -. "Dispatch" .-> R & V & E & VA & RA SA -- "Push/Pop" --> TQ %% Output & Verification Flow Tactical -- Exploit PoC --> VA VA -- Confirmed --> RA Validation <--> Blackboard %% Traffic Flow Toolset -- Traffic --> P_Svc VA -- Verification Traffic --> P_Svc P_Svc -- Request --> TWA TWA -- Response --> P_Svc%% Style Definitions style U fill:#f8f9fa, stroke:#212529, stroke-width:2px style SA fill:#eef6ff, stroke:#007bff, stroke-width:2px style Control_Plane fill:#ffffff, stroke:#007bff, stroke-width:2px, stroke-dasharray: 5 5 style Blackboard fill:#f8f9ff, stroke:#5c6bc0, stroke-width:2px style TQ fill:#ffffff, stroke:#5c6bc0, stroke-width:1px style FG fill:#ffffff, stroke:#5c6bc0, stroke-width:1px style HT fill:#ffffff, stroke:#5c6bc0, stroke-width:1px style Tactical fill:#ffffff, stroke:#333333, stroke-width:2px, stroke-dasharray: 5 5 style R fill:#f0f7ff, stroke:#2196f3, stroke-width:1px style V fill:#f3e5f5, stroke:#9c27b0, stroke-width:1px style E_Unit fill:#fff5f5, stroke:#f44336, stroke-width:1px, stroke-dasharray: 2 2 style E fill:#ffebee, stroke:#f44336, stroke-width:1px style SQLI fill:#ffffff, stroke:#f44336, stroke-width:1px style XSS fill:#ffffff, stroke:#f44336, stroke-width:1px style ETC fill:#ffffff, stroke:#f44336, stroke-width:1px style Toolset fill:#fffaf0, stroke:#fb8c00, stroke-width:2px style P_Svc fill:#e8f5e9, stroke:#4caf50, stroke-width:2px style TWA fill:#f5f5f5, stroke:#757575, stroke-width:2px style Validation fill:#ffffff, stroke:#28a745, stroke-width:2px, stroke-dasharray: 5 5Design Philosophy Some proprietary details have been omitted. Multi-Agent System: We adopted a multi-agent system in which multiple specialized agents collaborate to execute a pentest. Some notable OSS projects, such as westonbrown/Cyber-AutoAgent, are built as a single agent. There is no definitive answer as to which approach is better, but we chose a multi-agent system — accepting the added complexity of managing multiple components — for the following reasons:XBOW, a leading project in this space, also adopted a multi-agent system. Easier per-phase tuning: Models, system prompts, tools, and guardrails can be flexibly designed for each phase. Mechanical false-positive checkers can be implemented in an optimized form for each phase. Certain phases benefit from unique state management and forgetting mechanisms.Findings Graph as SSOT (Single Source of Truth): The Findings Graph serves as the SSOT for centrally managing intelligence throughout the entire attack chain. This mechanism provides the following benefits:Ensuring context consistency across the entire attack chain. Mitigating the risk of forgetting critical information. Improving token efficiency through mechanical extraction of task-relevant subgraphs.The Findings Graph is not merely a knowledge base — it is a critical foundation that plays a central role in orchestration. As the Findings Graph infrastructure encapsulates our core proprietary know-how, its details remain confidential. System-Level Guardrails: Since full automation without HITL (Human-in-the-Loop) is a core concept, guardrails that ensure safe operation without human oversight are essential. In addition to prompt-level guardrails, a man-in-the-middle proxy (L7 Egress Gateway) functions as a system-level guardrail. This prevents unintended attacks and excessive requests, ensuring the safety of the fully autonomous pentest AI agent. Field Test HackerOne We tested against a total of 30 target URLs under the following conditions:Time limit: 1 hour Input: URL only (black-box testing)The results were as follows:VDP (Vulnerability Disclosure Program) 3 vulnerability reports triaged at the U.S. Department of Defense (DoD). Achieved #86 worldwide in the 90-day leaderboard.BBP (Bug Bounty Program) 3 submissions (duplicates).Having confirmed the agent's effectiveness against production systems, we have concluded our HackerOne experiments and are now focusing on developing proprietary benchmarks and tuning, as described below.Pro Bono PoC With the cooperation of a partner company, we conducted a fully autonomous pentest as a pro bono PoC using our in-development pentest AI agent. In a gray-box test with minimal information provided (URL and credentials), the agent successfully discovered multiple exploitable vulnerabilities, including Stored XSS. Benchmark PortSwigger Academy Lab In experiments conducted with a 1-hour time limit, the agent solved 84% of the labs as of 2025/12/2.Proprietary Benchmark To fully leverage the advantage of easier per-phase tuning, a benchmark for evaluating tuning changes is essential. While the XBOW Benchmark and PortSwigger Academy Lab are useful for evaluating the entire attack chain, we believe they are not well-suited for per-phase tuning. To address this, we are developing a proprietary benchmark for per-phase performance improvement and cost optimization. Details will be shared in a future blog post. Conclusion We developed a fully autonomous pentest AI agent for web applications and achieved promising results in our first field test. We will continue R&D to automate more advanced attack chains and expand the range of supported targets. We are looking for companies willing to participate in proof-of-concept testing of our pentest AI agent. If you are interested, please reach out to us.