AI penetration testing means using AI agents, with the owner's written permission, to probe a website or system for weak spots the way an attacker would. It goes further than a vulnerability scanner, which only matches known flaws. Most small businesses need a scanner and a checklist first, and an AI pentest second.
What is AI penetration testing?
AI penetration testing is the use of AI agents to probe your own website or systems for weak spots, the way an attacker would, with your permission. NIST describes a penetration test as security testing in which evaluators imitate real-world attacks to find ways around a system’s defences. The AI version hands part of that job to an agent, which is an AI system that can run tools and take steps on its own.
The test itself is old. What AI changes is how much of the routine work a person has to do by hand.
How is an AI pentest different from a plain scanner?
A vulnerability scanner checks for known problems, while a penetration test tries to use them. NIST’s testing guide, SP 800-115, covers both as techniques for finding weaknesses and confirming whether security requirements are met. The table compares the three approaches in general terms.
| Vulnerability scanner | Manual pentest | AI-assisted pentest | |
|---|---|---|---|
| What it does | Matches your software and settings against known flaws | A human tester imitates an attacker and chains weaknesses | An AI agent does much of the probing, with a human setting scope and reviewing |
| Speed | Minutes to hours | Days to weeks | Faster than manual, varies by tool |
| Judgement | None, a list of matches | High, human reasoning | Some, but results need checking |
| Typical output | A list of findings | A written report with attack paths | A report, often needing human validation |
| Best for | Routine checks, every month or so | Important systems, compliance | Frequent checks between manual tests |
The speed and judgement rows are CoSA Lab’s general summary, not figures from a standard.
What does an AI pentest actually check?
The checks follow the same ground as a manual test. The OWASP Web Security Testing Guide is the widely used reference for web application testing, and its current stable release is version 4.2. A test usually covers:
- Which pages, logins and services the site exposes.
- Whether software versions have known flaws.
- How logins, sessions and permissions are handled.
- Whether forms and inputs accept things they should reject.
CoSA Lab does not describe attack methods here. The point for an owner is to know what the report should cover.
Can AI really find security flaws?
Yes, with limits. Anthropic’s research on AI for cyber defenders, published in September 2025, says AI models have become practically useful for finding vulnerabilities in code. It gives benchmark results and says teams using AI models found some previously unknown flaws in real software. Anthropic also says these abilities are still early and do not match experienced security professionals.
In practice, treat an AI report as a list of leads. A person should confirm each finding before anyone changes anything.
Does a small business need an AI penetration test?
Most small businesses should start with simpler steps. A site with out-of-date software or weak admin logins does not need a pentest to reveal the problem. Our website weak-spot checklist covers those basics with free tools.
A pentest becomes worth the cost when:
- Your site takes payments or stores customer data.
- You run a custom application, not just standard WordPress or Shopify.
- A customer, insurer or regulator asks for test evidence.
- You have fixed the basics and want a second opinion.
How do I run one safely?
Get written permission, define the scope, test a copy where possible, and keep logs. Only test systems you own or have written authorisation to test. The full steps are in using AI agents to test your own site, safely. Background on why attackers use the same approach is in how AI attackers work, and the term is defined in the AI pentesting glossary entry.
What should I ask a provider before hiring one?
Ask for the scope, the method, the report format and who reviews the AI’s output, before you sign anything. A good provider can answer each of these plainly:
- Scope: Which domains, applications and accounts are included, and which are off limits.
- Human review: Whether a named person checks every finding, since AI agents can report flaws that do not exist.
- Safety limits: Whether the agent works from read-only access and a staging copy, and how its actions are logged.
- Reporting: Whether findings come with plain-language fixes ranked by risk, plus evidence you can check.
- Retest: Whether a second run after your fixes is included.
NIST’s SP 800-115 treats planning and reporting as part of the testing process, not extras. A provider who skips them is selling a scan under another name.
What can an AI pentest miss?
An AI pentest can miss problems that depend on how your business works. A scanner or agent sees technical settings. It does not know that a discount code should work only once, or that a staff member can see another customer’s order. Anthropic says its own AI defender tools do not yet match experienced security professionals, and that is a fair general caution.
Treat a clean report as a sign that the tested items passed, not that the site is secure.
A clean result is still useful. It gives you a dated baseline to compare against after your next update.
Common questions
What is the difference between a vulnerability scan and a penetration test?
A vulnerability scan automatically checks a system against a list of known flaws and reports matches. A penetration test goes further: a tester tries to use weaknesses, often chaining several together, to see how far an attacker could get.
Is AI penetration testing legal?
It is legal on systems you own or have written permission to test. Testing someone else's system without permission can be a crime in most countries.
How much does a penetration test cost?
Prices vary widely by scope and provider, and the sources used here give no figures. Ask for a fixed scope and quote before work starts.
Sources
- NIST: Technical Guide to Information Security Testing and Assessment (SP 800-115)(csrc.nist.gov)
- NIST CSRC Glossary: penetration testing(csrc.nist.gov)
- OWASP: Web Security Testing Guide(owasp.org)
- Anthropic: Building AI for cyber defenders(anthropic.com)
