Benchmark
Vulkro is measured two ways. First, head to head against six other scanners on one ground-truth corpus, on the same machine, on the same day, with the same scoring code. Second, on the OWASP Benchmark, the industry-standard synthetic test suite that anyone can rerun. Both runs are at vulkro 0.25.3.
Every bug in the head-to-head corpus is pinned to published provenance in a real project (the provenance rule below): a CVE or GHSA advisory where one exists, and otherwise the upstream project's own in-source vulnerability marker or documented challenge, which is how deliberately-vulnerable corpora such as Juice Shop, NodeGoat and DVGA identify their planted bugs. A finding counts only within five lines of the catalogued bug. Every table on this page, including the comparisons we lose, is scored that way. The methodology is published in full; you can check any number against the advisory it cites.
Headline results
The four tools that scan more than one language, rolled up over the scored
corpus: 83 catalogued bugs (one PHP repo, DVWA, is excluded from the roll-up
totals; Vulkro's PHP analysis is single-function, so this roll-up scores the
cross-file languages, and the scorecard still publishes DVWA's per-repo row and
states the exclusion). Vulkro is scored at --min-confidence high, the recommended CI
setting; the other tools' findings are never confidence-filtered (see
Methodology for why).
Read the precision row as exactly this and nothing more: precision against the catalogued-bug set of this curated corpus, at high confidence. It is not a claim about the share of all findings that are real on an arbitrary repo; the "what this precision is measured over" note below the table states the difference explicitly, and every figure regenerates from the committed scorecard.
| Vulkro | Tool A 2.0.2 | Tool B 1.169.0 | Tool E 1.25.0 | |
|---|---|---|---|---|
| Precision | 0.62 | 0.37 | 0.53 | 0.53 |
| Recall | 0.40 | 0.52 | 0.20 | 0.20 |
| F1 | 0.49 | 0.43 | 0.30 | 0.30 |
| Catalogued bugs found (of 83) | 33 | 43 | 17 | 17 |
| False positives | 20 | 72 | 15 | 15 |
| TP / FP / FN | 33 / 20 / 50 | 43 / 72 / 40 | 17 / 15 / 66 | 17 / 15 / 66 |
Why the other tools are not named. They appear as Tool A through Tool F, here and in every table below. That is site policy: we do not put other vendors' names on our own pages. The labels are stable across this page, and each tool keeps the pinned version number it ran at. State the cost plainly: with the names withheld you cannot take a row from this page and go check it against a named product yourself. What is not withheld is everything that decides what the numbers mean. The corpus and its provenance rule, the matching rule, the four buckets, the scoring formulas and the exact commands the harness runs are all published below, and the rows Vulkro loses are printed at the same size as the rows it wins. We do not publish the mapping, and the order of the labels carries no information.
How to read it:
- Precision and F1 lead the multi-language field. When Vulkro fires at high confidence it is right 0.62 of the time, the highest precision of any multi-language tool in the run, and its F1 of 0.49 is the highest as well.
- Recall is not the lead, and we say so. Tool A finds more of the catalogued bugs than Vulkro does (43 vs 33). It pays for that recall with 72 false positives to Vulkro's 20, roughly 3.6x the noise, which is why Vulkro still wins precision and F1. If you want the most catalogued bugs found regardless of noise, Tool A finds more here.
- Against the free community-ruleset lane. Vulkro finds 33 catalogued bugs to Tool B's 17 (about 1.9x) at higher precision (0.62 vs 0.53). Tool B and Tool E score identically because both lanes run the same live community security ruleset; the point of the Tool E lane is comparing engines on identical rules.
- What this precision is measured over. These figures count a finding against the catalogued-bug set. Vulkro emitted 266 total findings in this run, so the share of ALL findings that map to a catalogued bug is lower than 0.62. That is expected for a scanner that also reports IaC, secrets and dependency issues the corpus does not catalogue, but it is not the same statistic and we do not present it as one.
- Java is now in the roll-up. WebGoat and java-sec-code (both Java) are scored and included in these totals; Vulkro finds 4 of 4 catalogued bugs on WebGoat and 3 of 3 on java-sec-code. Only DVWA (PHP) is excluded from the totals: Vulkro's PHP analysis is single-function, this roll-up scores the cross-file languages, and DVWA's per-repo row is still published below. See Java and Spring.
OWASP Benchmark
The head-to-head above is our own corpus. The OWASP Benchmark is not: it is a public, standardized suite of 2,740 Java test cases with published ground truth, maintained by the OWASP Benchmark Project, that any scanner can be run against and scored the same way. We measured vulkro 0.25.3 against it on 2026-09-08 with the project's own average-of-categories scoring method.
| metric | vulkro 0.25.3 |
|---|---|
| OWASP Benchmark score (avg of per-category Youden) | 37.73 |
| Pooled true-positive rate (recall) | 69.6% |
| Pooled false-positive rate | 29.9% |
The score is the average across the eleven vulnerability categories of the Youden index (true-positive rate minus false-positive rate); 100 is a perfect scanner and 0 is a coin flip. Vulkro's 37.73 sits at the top of the band of reference scanners the OWASP Benchmark Project publishes on its own scorecard, whose scores run from 0 up to about 39; the single published tool that scores marginally higher does so at a false-positive rate near 58 percent, against Vulkro's 30 percent. You can look up the published reference scores in the OWASP Benchmark repository and reproduce Vulkro's row with the steps in our benchmark harness.
Per category, Vulkro is near-perfect on some and weak on others, and both are on the record:
- Strong: insecure cookie flags (Youden 100), weak randomness (92), weak hashing (69), broken crypto (55).
- Weak: command injection, trust-boundary, and XPath injection score in the single digits to low twenties, driven by taint that does not reach the sink or by false positives on the synthetic patterns.
One honest limitation: the OWASP Benchmark is synthetic Java microcode built to exercise taint patterns, not a real application. A high score there means the taint engine handles the planted patterns well; it does not by itself predict real-world precision, which is why we publish the real-project head-to-head above as well.
The corpus, version 2026-07-05
The Tier 1 corpus is 15 public repositories, pinned by SHA: 12 deliberately-vulnerable apps with 90 catalogued bugs, plus 3 clean framework controls (express, fastapi, flask) that must produce zero findings. 83 of the 90 bugs are in the supported-language roll-up above; the other 7 live in the PHP repo (DVWA), which stays out of the roll-up.
We grew our own ground truth from 69 to 90 bugs in July 2026 and published the recall drop that came with it. The growth batch added two repos chosen to be hard for us (DVGA, a Python GraphQL app, and govwa, a Go app, both of which score low Vulkro recall today) and previously-missed bugs in existing repos. The corpus is versioned; every change is in the changelog.
The provenance rule. Every catalogued bug must be independently verifiable and
must cite the repo's own vulnerability documentation as the primary source: its
README or solutions doc, an in-source marker (juice-shop's vuln-code-snippet
comments, NodeGoat's tutorial pages), or a published CVE / GHSA pinned to the
commit. Each entry is then confirmed by reading the sink at the pinned SHA.
Selection is not biased toward what Vulkro detects: catalogued bugs Vulkro misses
stay in the corpus as legitimate false negatives. That honesty is the point of
the benchmark.
[[vuln]]
file = "app/routes/contributions.js"
line = 32
class = "command-injection"
cwe = "CWE-94"
note = "eval(req.body.preTax) - A1 SSJS injection (source comment line 31)"
Methodology
Ground truth and matching
Each catalogued bug is a (file, line, class) tuple. A finding matches a
ground-truth entry iff the files are equal and the finding line is within 5 lines
of the catalogued line (+/-5). Each ground-truth entry can be claimed by at
most one finding (greedy nearest), so over-firing on the same bug does not
inflate the score.
The four buckets
| bucket | meaning |
|---|---|
| TP | Tool fired on a catalogued bug (same file, line within +/-5). |
| FP | Tool fired in a file we catalogued, but at no catalogued bug line. |
| FN | Catalogued bug, no tool finding fired against it. |
| OOS | Tool fired in a file we did not catalogue. Neither rewarded nor penalized: we do not claim to know whether those files are clean or just unannotated. |
The OOS bucket is what separates "noisy" from "broad": a tool that fires thousands of times outside the catalogued files is neither punished for it nor credited with it.
Confidence filtering: Vulkro only, and why
Vulkro is scored at --min-confidence high for the headline (its recommended CI
setting) and at low for the labeled max-recall run. The other tools' findings
are never confidence-filtered, because their confidence ladders are not
directly comparable to ours: dropping another tool's "low confidence" findings
using our idea of what "low" means would silently rig the comparison. Every other
lane runs at its own defaults instead.
Metrics
- precision = TP / (TP + FP): when the tool fires, how often is it right?
- recall = TP / (TP + FN): of the catalogued bugs, how many did it find?
- F1 = 2 x (precision x recall) / (precision + recall): the standard combined score.
Per-language specialists: the fair-fight rule
Tool F only scans Python, Tool D only Go, Tool C only JavaScript. Rolling a single-language tool up over a mixed corpus would manufacture zeros for it, so the specialists never enter the roll-up. Instead, each specialist is scored only over the repos matching its language, and Vulkro is re-scored over the same repo subset. Every comparison below is a fair fight, and the one Vulkro loses is printed just as large.
Tool F vs Vulkro, 7 Python repos
| Tool F 1.9.4 | Vulkro (same repos) | |
|---|---|---|
| Precision | 0.56 | 0.84 |
| Recall | 0.43 | 0.38 |
| TP / FP / FN | 18 / 14 / 24 | 16 / 3 / 26 |
On its home language, Tool F finds 18 of the catalogued Python bugs with 14 false positives; Vulkro finds 16 with only 3. Vulkro is far more precise and marginally lower on recall.
Tool D vs Vulkro, 1 Go repo: Tool D wins
| Tool D 2.27.1 | Vulkro (same repo) | |
|---|---|---|
| Precision | 0.20 | 0.11 |
| Recall | 0.75 | 0.25 |
| TP / FP / FN | 3 / 12 / 1 | 1 / 8 / 3 |
On our one Go repo (govwa), Tool D finds 3 of 4 bugs; Vulkro finds 1, with 8 false positives. Go detection depth is a known gap we publish. The misses stay catalogued as false negatives rather than being quietly dropped. If Go is your primary language, Tool D is free and it beats us there today.
Tool C vs Vulkro, 4 JS/TS repos
| Tool C 0.4.3 | Vulkro (same repos) | |
|---|---|---|
| Precision | 0.50 | 0.64 |
| Recall | 0.33 | 0.30 |
| TP / FP / FN | 10 / 10 / 20 | 9 / 5 / 21 |
Vulkro is more precise; recall is close, and Tool C edges it by one catalogued
bug on this subset. One caveat the per-repo rows keep visible: Tool C targets
plain JavaScript, so on the TypeScript repo (juice-shop) it sees almost nothing in
.ts files.
Max-recall mode
Drop the confidence threshold (--min-confidence low) and Vulkro trades precision
for recall. The published scorecard labels this run as max-recall:
| metric | Vulkro max-recall | Vulkro headline (high) |
|---|---|---|
| Precision | 0.16 | 0.62 |
| Recall | 0.73 | 0.40 |
| TP / FP / FN | 61 / 312 / 22 | 33 / 20 / 50 |
Max-recall finds 61 of the 83 roll-up bugs at the cost of 312 false positives. It
is the right setting for a one-shot audit where a human will triage everything
once; --min-confidence high is the right setting for CI. We publish both so the
trade-off is yours to make, not ours to hide.
Where Vulkro misses
The corpus is built to keep our failures on the record:
- GraphQL resolver taint. DVGA (Damn Vulnerable GraphQL Application) has 6 catalogued bugs reached through GraphQL resolvers. Vulkro finds 0 of them. Tracing taint through resolver dispatch is an open detection gap.
- Go. govwa has 4 catalogued bugs; Vulkro finds 1, Tool D finds 3 (see the fair-fight table above).
- PHP. DVWA (7 bugs, PHP) is excluded from the roll-up totals: Vulkro's PHP analysis is single-function and this roll-up scores the cross-file languages. Its per-repo row is still published in the scorecard.
By bug class, the headline-run misses concentrate in: command injection (11), SQL injection (10), open redirect (6), XSS (5), plus smaller counts of insecure deserialization, IDOR, hardcoded secrets, insecure cookies, path traversal and SSRF. The full FN-by-class table is regenerated in every scorecard.
Speed
Wall clock for the whole corpus, one full end-to-end process invocation per tool per repo. Measured 2026-09-08 on an Apple M5 Pro, 24 GB RAM, macOS 26.5.1.
| tool | whole-corpus scan time |
|---|---|
| Tool D 2.27.1 | 0.8s (1 Go repo only) |
| Tool F 1.9.4 | 4.2s (7 Python repos only) |
| Tool C 0.4.3 | 25.1s (4 JS/TS repos only) |
| Vulkro 0.25.3 (high confidence) | 37.1s |
| Vulkro 0.25.3 (default) | 37.5s |
| Tool B 1.169.0 | 48.4s |
| Tool E 1.25.0 | 59.8s |
| Tool A 2.0.2 | 392.7s |
The honesty notes that come with that table:
- This reference is a single cold run per tool per repo (
--timing-runs 1), because the 7-tool matrix makes median-of-3 a very long session. For publishable single-tool comparisons the harness supports--timing-runs 3and we recommend it. - Repos are pre-cloned to local disk before timing; clone time is excluded. Every sample includes process start, scan, and report serialization.
- Tool-internal result caches are disabled where the tool supports it (Tool A runs with its cache forced off), so every sample is a cold scan. Tool A's incremental cache makes its repeat scans on unchanged code much faster than shown here.
- The specialists only scan their matching-language repos, so their totals cover only that subset. A 0.8s single-language total is not comparable to a 37.1s whole-corpus total and the table says so.
- Wall clock is machine-dependent. Compare the ratios between tools on the same machine, not absolute seconds across machines.
How the numbers are produced
You do not have to take any number on this page on faith. Two things make it checkable. First, every catalogued bug cites a public CVE or GHSA advisory in a real project (the provenance rule above), so you can verify any single entry against its source. Second, the OWASP Benchmark is fully public: clone it, scan it, and score it with the project's own method to reproduce Vulkro's 37.73 row yourself.
What "reproducible" means for the head-to-head run itself: same Vulkro binary + same corpus SHAs + same scoring code = same TP/FP/FN counts, every run, on any machine. Known sources of drift:
- The shared community ruleset is not pinned. The ruleset behind the Tool B and Tool E lanes is served live by its own registry and changes over time, so those two counts can move without any change on our side.
- Other tools' versions. Newer releases change rules. The scorecard records the versions it ran; compare like with like.
- Hardware moves speed numbers. Compare ratios, not absolute seconds.
One honest limitation: the head-to-head harness and the pinned corpus live in our private repository today, so you cannot run those commands yourself yet. We publish the method, the per-bug advisories, and the comparisons we lose. The OWASP Benchmark run, by contrast, is fully reproducible from public sources.
Tool versions in the published run
| tool | version | note |
|---|---|---|
| Vulkro | 0.25.3 | scored at --min-confidence high (headline) and low (max-recall) |
| Tool A | 2.0.2 | SAST lane only, result cache disabled |
| Tool B | 1.169.0 | community security ruleset, fetched live, not pinned upstream |
| Tool C | 0.4.3 | JS/TS repos only; bundles a third-party pattern engine |
| Tool D | 2.27.1 | Go repos only (specialists section) |
| Tool E | 1.25.0 | same live community security ruleset as the Tool B lane |
| Tool F | 1.9.4 | Python repos only (specialists section) |
Tools not included
One authenticated commercial scanner is deliberately left out of the published runs. It cannot run without an authenticated account, and publishing benchmark results may be restricted by its terms of service (a check we have not completed). The harness keeps a fully working runner for that lane: enable it and authenticate to produce the numbers locally for yourself, after reading its current terms.
Two DAST products are also out. They cannot scan a static code directory at all and exist only as documented stub lanes in the harness.
Caveats: what these numbers do not tell you
- The head-to-head corpus is deliberately-vulnerable code. Real-world performance varies; deliberately-vulnerable apps are denser in bugs and lighter in framework noise than production code. The 3 clean framework controls (express, fastapi, flask) are the noise-floor check inside the corpus.
- The OWASP Benchmark is synthetic. A high score there measures taint handling on planted Java patterns, not real-app precision.
- Class matching matters. A finding under the wrong class does not count as a TP even when the line is right. This penalizes every tool roughly equally.
- Ground truth is a floor, not a census. The catalogued bugs are the bugs we could verify by provenance; the OOS bucket exists precisely because we do not pretend the rest of every file is clean.
- A benchmark is one input. Run the tools on your own code.
Related
- Confidence model - how
HighvsMediumvsLowis computed from evidence weights. - Detection coverage - per-language detector support.
- Reproducing in CI - running the benchmark as part of every PR.