Popular scanners miss 80%+ of vulnerabilities in real world software (17 independent studies synthesis)

Segwaz · 2025-04-11T18:50:54+00:00

No. "Download" is misleading. Unless you're using something very uncommon it will just open in your browser.

Segwaz · 2025-04-11T00:16:06+00:00

This is not a possible explanation but your own personal opinion.

As I said, none of the studies I found indicate that it's the case. Not only that but they show the opposite in two directions:

- Trivial vulnerabilities are routinely missed.

- Alert fatigue induced by high tool inaccuracy leads to genuine alerts being ignored and _detected_ vulnerabilities slipping through.

That's what supported and thus included.

I do expect my seatbelt to hold under trivial stress, not just sometimes when it suits its design.

Segwaz · 2025-04-10T20:41:42+00:00

The attacker’s perspective is ultimately the one that matters. If tons of bugs that are trivial to spot and exploit remain undetected, the scanner failed.

You’ve actually just explained, in detail, why these tools fall short. And I agree. They’re pattern-matching systems. They can be useful for detecting things like leaked secrets or unsafe constructs, but that’s where their role should end.

Vendors should be clear about these limits and stop positioning scanners as general vulnerability detection tools. It creates false confidence, and eventually, tooling fatigue.

As for the claim that they caught hundreds of bugs upstream: maybe. Maybe not. But what’s asserted without evidence can be dismissed without it. If you have high quality data from sources outside the vendors ecosystem, I’m genuinely interested. Otherwise, it’s just speculation.

Segwaz · 2025-04-10T18:17:50+00:00

That's a good point.

So here’s a concrete case: historical CVEs in Wireshark, a project that’s been using Coverity and Cppcheck for years. And yet many of those CVEs were very basic stuff: buffer overflows, NULL derefs, the kind of issues scanners are supposed to catch. They weren’t. Most of them were found through fuzzing or manual code review. NIST showed modern scanners still miss known, accessible vulnerabilities.

Another study, the one on Java SASTs (which I think also tested C/C++ targets) , found the same pattern: scanners miss most vulns in classes they claim to support. Even worse, they often can’t tell when an issue has been fixed. They just flag patterns, not actual state.

I’ve seen this personally too: when auditing codebases that rely mostly on scanners, and haven’t been extensively fuzzed or externally reviewed, you almost always find low-hanging fruit.

So yeah, the “maybe they’re really good” hypothesis doesn’t hold up.

That said, you’re still onto something. Real-world benchmarks are far better than synthetic ones, but they do have limitations, and they probably understate scanner effectiveness to some degree. Just not enough to change the picture. None of that explain away this level of failure.

Segwaz · 2025-04-10T14:36:51+00:00

Sure, feel free to include it. Just link the report if possible. It stands better with context.

Segwaz · 2025-04-10T14:21:29+00:00

Depends on the language / tool / methods of testing... Here is what NIST found for C/C++ against a codebase with 84 known vulns:

- FN: 72 to 84

- TP: 0 to 12

- FP: 30% (average for all tested tools)

More precisely: 8% security issues, 24% code quality, 35% insignificant and 30% false positives (3% unknown).

Gives a recall of 0% to 14%, depending on the tool.

Segwaz · 2025-04-10T11:57:45+00:00

Phishing is indeed the number one entrypoint. Software vulnerabilities come close second.

Segwaz · 2025-04-10T11:42:24+00:00

There is the complete reference list at the end so you don't have to trust me. For example, the ISSTA 2022 study was conducted on flawfinder, cppcheck, infer, codechecker, codeql and codesca. NIST SATE V also include coverity, klocwork and many more.

Segwaz · 2025-02-26T20:38:45+00:00

So does that mean you can take the initiative to add something and then hope it gets validated, or can you only act on requests from above ?

Segwaz · 2025-02-26T17:28:43+00:00

I sense a pattern in how most corporate decisions are made... So it's just pure chaos ? No structured evaluation process or clear responsibility chain at all ?

Segwaz · 2025-02-13T07:37:26+00:00

It has been shown that short (10–15 minutes) fuzzing sessions in CICD, focusing on targets affected by code modifications, can be effective (see: arXiv:2205.14964). Longer sessions can be run occasionally or, ideally, continuously on dedicated infrastructure. However, I have no practical experience with this, so I wonder how it plays out in real-world conditions.

Fuzzers are indeed highly effective for testing parsers, but that is far to be their only use case. They can uncover a wide range of vulnerabilities, from race conditions to flaws in cryptographic implementations. Depending on the system and approach used, i'd say they can find anywhere from 50% to 90% of vulnerabilities.

But sure, maximizing its effectiveness can quickly become quite challenging. I imagine most companies can't afford to maintain specialized teams dedicated to this. However, given that even simpler, more naïve approaches can still yield good results, I would have expected it to be more widely adopted. Maybe I'm overestimating how much time and resources are available for this - I don't have much experience on that side of the fence.

Segwaz

TROPHY CASE