I wonder what everyone’s experience with LLMs for pentesting. I’ve been using GPT 5.6 Sol (although got kicked from the cyber program) and GLM 5.2.
I’ve been getting good results with minimal input, smarter models once they have completed mapped an application and functionality, they’re pretty good at finding vulns. I can confidently say they can replace Nessus.
I think the main shift right now is getting smaller models (I’m talking 2B, 3B, and so on) specialized in pentesting with given knowledge bases with test cases.
I understand using a model to drive offensive tools, but I’m curious what value it provides over Nessus in terms of positively identifying unpatched vulns. Like I’d have to imagine that’s a lot of tokens for an ultimately deterministic process.
It comes down to data analysis, when the scan results are way to big. it’s been 2 years since I last used Nessus, I’m not sure if they improved it, but analyzing a 10,000 host nessus scan is overwhelming with the usual view.
Replacing Nessus feels like a bit of a stretch if you look at the costs versus benefits. I haven’t worked on the AD/network pentesting side in a while, but as far as applications go, they certainly can be helpful in reducing the tedium of sifting through source code and following data flows. More often than not, however, if I leave an LLM unchecked (e.g. just letting it use Burp MCP with manually approving requests), they still produce quite a bit of noise either by claiming false vulnerabilities or overstating severity ratings.
If you have a very focused question you want surface level digging on, like “Review the source code and identify all of the endpoints along with what authentication/authorization controls are applied to it”, it can save some hours. If you just say “go find an 0day”, it’s quite wasteful.
It’s a business decision at the end of the day, if you trust ZDR policies of your provider and the client allows it, it’s fine. Some clients even want you to use AI.
Nessus is not cheap at all
For it to be a Nessus replace, you don’t let the AI decide what to do in each single run, there must be procedures/skills for each type of analysis that will let it know what it should test for.
Can you explain what value the model provides in the discovery of vulns over Nessus? I think I understand the reporting/triaging aspect, but I think I’m bumping at the idea of wholly replacing a deterministic vuln scanner with a model that needs so much handholding.
Also, on cost: I’m very wary of building any long-term process on a technology that will necessarily either a) explode in cost or b) cease to exist as a ubiquitous service. There is not a third outcome to the economic realities.
discovery of vulns comes from tools (nmap, nuclei, feroxbuster, etc..) that you give to the agent, not the agent itself. That’s the only way to be deterministic.
Now, why let an AI agent run the tools instead of yourself? and what advantage has that over Nessus? Networks are weird, there are blockers always, firewall on the middle that you need to bypass, ports that don’t really have anything and you get flooded on Nessus, if one technique doesn’t work, it can execute another one that it has in its knowledge base.
Also one really important thing that I see most people skipping, you need to keep an activity log of EVERYTHING that either AI keeps or you automatically collect with whatever technique you want. Having this let’s the AI review what it did well, wrong and so on for self improvement reducing the handholding to a minimum, as well as, letting you review the work the agent did.
You’re completely right, but the industry is on this path, and as much as we can hate it, we can’t stop it. That said, I don’t believe that AI will disappear; maybe one provider will, heck Anthropic can disappear tomorrow and that’s why you should be provider agnostic, you need to be able to take any provider.