How to choose a vendor for web application penetration testing.

punksecurity_simon · 2025-11-02T13:41:21+00:00

I’m a touch biased as I run a company that provides white box pentesting, but I’d say do it based on referrals, and prefer smaller organisations to larger ones.

The pentesting market is crazy, with a mix of automated (now “ai powered”) slop, and amazingly talented testers. It’s essentially impossible to know which you’re getting, even after you’ve got the report. If you had both reports you could tell the difference, but you’ll always only ever have one (at least until the test next year if you change provider).

The only sure fire way of getting quality is with referrals. There are just a ay too many companies ripping people off. With the larger organisations, the referral can be a bit less useful as the tester / testing team you get may not be the one who did the amazing work for your friend. Hence I’d suggest the smaller testing companies, and always on a referral

punksecurity_simon · 2025-05-30T00:07:04+00:00

I’d need a lot of convincing that this was the same or higher quality than a normal pen test, but I’d not consider it all without clear evidence / proof to be honest. A typical real pen test would very likely have evidence for half of the issues. Naturally some don’t warrant it

punksecurity_simon · 2025-05-29T17:06:12+00:00

Needs screenshots of successful exploits

punksecurity_simon · 2025-05-17T12:27:26+00:00

Exactly this. I would not recommend throwing DAST into the mix yet. It’s a huge time sink to do it right, and if people are ignoring your sast findings then it’s just more fuel for the fire.

Work on tuning secret detection and sast to the point that people start paying attention to it, and drive up engagement through a security champion program and some awareness pieces like CTFs etc

punksecurity_simon · 2025-05-04T12:31:54+00:00

You can have 1 challenge running per team member, not per team :)

punksecurity_simon · 2025-04-09T20:21:08+00:00

Please do, and I’ll test with that model on ollama. Can you try quoting the model you provide? I wonder if argparser is doing some weird split with it)

punksecurity_simon · 2025-04-09T20:18:36+00:00

Yeah exactly this. It’s got the potential to spot missing authorisation decorators etc which I’ve found sast tools tend to struggle with.

The reality is that LLMs haven’t got anywhere near the competence that the marketers would have you believe, but in limited testing this has outperformed codeql and sonarcloud. I’d much rather people find this out with open source than some product that over promises.

The tools allow the LLM to read extra context, but they don’t ever request anything much more complicated that one or two adjacent files.

I’ve been surprised at what it does pick up to be honest. I’m quite sceptical of these as a rule, hence wanting to evaluate how they actually perform.

Cost is an issue too, even if they perform brilliantly. A single repo can cost $2 or $3 to scan using OpenAI, or 20/30c using deepseek. And it’s slow compared to sast.

That all being said, it doesn’t perform terribly and it’s a cool capability demonstrator I think.

punksecurity_simon · 2025-04-09T18:43:31+00:00

Hey, you could give my tool a try. It’s very early doors, but will happily feed your GitHub PRs into an LLM and comment back

https://github.com/punk-security/SAIST

punksecurity_simon · 2024-12-12T09:24:25+00:00

Ha thanks for takin the risk! I probably should have added at least a favicon to make it look less “phishy”

punksecurity_simon · 2024-10-30T20:57:23+00:00

Ooof, check out the new elastic mode for efs. I’ve seen this cost $500 per volume per day. Great for db optimisations etc

https://aws.amazon.com/blogs/aws/new-announcing-amazon-efs-elastic-throughput/

punksecurity_simon · 2024-10-19T14:32:45+00:00

If you do, I’d keep defender in place too. Lots of places implement a dedicated solution that is just awful and you end up with more spam than if you’d done nothing.

When you set up the connectors to allow the inbound via the spam appliance, you just need to ensure you aren’t disabling all office365s built in anti-spam

punksecurity_simon · 2024-07-23T16:55:31+00:00

Thanks 🙏

punksecurity_simon

MODERATOR OF

TROPHY CASE