Skip to main content

Benchmark

Vulkro is measured against six other scanners on one public ground-truth corpus, on the same machine, on the same day, with the same scoring code. Every table on this page is generated by the published harness, including the comparison we lose. One command reruns the whole thing:

VULKRO="$(command -v vulkro)" bench/comparison/run.sh --tier1 \
  --tools vulkro,semgrep,opengrep,bearer,bandit,gosec,njsscan \
  --min-confidence high

Headline results​

The four tools that scan more than one language, rolled up over the 12 repos in the scored roll-up (76 catalogued bugs). Vulkro is scored at --min-confidence high, the recommended CI setting; competitor findings are never confidence-filtered (see Methodology for why).

Vulkro 0.15.0Semgrep CE 1.136.0Opengrep 1.25.0Bearer 2.0.2
Precision0.810.620.620.36
Recall0.580.200.200.49
F10.680.300.300.41
Catalogued bugs found (of 76)44151537
False positives109967
TP / FP / FN44 / 10 / 3215 / 9 / 6115 / 9 / 6137 / 67 / 39

How to read it:

  • Precision first. When Vulkro fires, it is right 0.81 of the time. That beats every multi-language tool in the run, at more than double their recall.
  • Against the free baseline. Vulkro finds 2.9x the catalogued bugs Semgrep CE finds (44 vs 15) at higher precision (0.81 vs 0.62). Semgrep CE and Opengrep score identically because both lanes run the same p/security-audit ruleset; the point of the Opengrep lane is comparing engines on identical rules.
  • Against the nearest rival. Bearer is the only tool close on recall (37 vs 44) and pays with 67 false positives to Vulkro's 10.
  • The PHP and Java repos are not hidden. All four tools also scan DVWA (PHP), WebGoat, and java-sec-code (Java); the per-repo rows are published in the scorecard. They are excluded from these totals for two different reasons. PHP is out of Vulkro's language scope. Java is now analysed - see Java and Spring - but that support shipped after this corpus was scored (these numbers come from a Vulkro 0.15.0 run), so the Java repos stay out of the roll-up pending a re-score. Bearer's wins on those repos are called out in Where Vulkro misses.

The corpus, version 2026-07-05​

The Tier 1 corpus is 15 public repositories, pinned by SHA: 12 deliberately-vulnerable apps with 90 catalogued bugs, plus 3 clean framework controls (express, fastapi, flask) that must produce zero findings. 76 of the 90 bugs are in the supported-language roll-up above; the other 14 live in the PHP and Java repos.

We grew our own ground truth from 69 to 90 bugs in July 2026 and published the recall drop that came with it. The growth batch added two repos chosen to be hard for us (DVGA, a Python GraphQL app, and govwa, a Go app - both score 0 Vulkro recall today) and 21 previously-missed bugs in existing repos. The corpus is versioned; every change is in the changelog.

The provenance rule. Every catalogued bug must be independently verifiable and must cite the repo's own vulnerability documentation as the primary source: its README or solutions doc, an in-source marker (juice-shop's vuln-code-snippet comments, NodeGoat's tutorial pages), or a published CVE / GHSA pinned to the commit. Each entry is then confirmed by reading the sink at the pinned SHA. Selection is not biased toward what Vulkro detects: catalogued bugs Vulkro misses stay in the corpus as legitimate false negatives. That honesty is the point of the benchmark.

[[vuln]]
file = "app/routes/contributions.js"
line = 32
class = "command-injection"
cwe = "CWE-94"
note = "eval(req.body.preTax) - A1 SSJS injection (source comment line 31)"

Methodology​

Ground truth and matching​

Each catalogued bug is a (file, line, class) tuple. A finding matches a ground-truth entry iff the files are equal and the finding line is within 5 lines of the catalogued line (+/-5). Each ground-truth entry can be claimed by at most one finding (greedy nearest), so over-firing on the same bug does not inflate the score.

The four buckets​

bucketmeaning
TPTool fired on a catalogued bug (same file, line within +/-5).
FPTool fired in a file we catalogued, but at no catalogued bug line.
FNCatalogued bug, no tool finding fired against it.
OOSTool fired in a file we did not catalogue. Neither rewarded nor penalized: we do not claim to know whether those files are clean or just unannotated.

The OOS bucket is what separates "noisy" from "broad": a tool that fires thousands of times outside the catalogued files is neither punished for it nor credited with it.

Confidence filtering: Vulkro only, and why​

Vulkro is scored at --min-confidence high for the headline (its recommended CI setting) and at low for the labeled max-recall run. Competitor findings are never confidence-filtered, because their confidence ladders are not directly comparable to ours: dropping a competitor's "low confidence" findings using our idea of what "low" means would silently rig the comparison. Each competitor runs at its own defaults instead.

Metrics​

  • precision = TP / (TP + FP): when the tool fires, how often is it right?
  • recall = TP / (TP + FN): of the catalogued bugs, how many did it find?
  • F1 = 2 x (precision x recall) / (precision + recall): the standard combined score.

Per-language specialists: the fair-fight rule​

Bandit only scans Python, Gosec only Go, njsscan only JavaScript. Rolling a single-language tool up over a mixed corpus would manufacture zeros for it, so the specialists never enter the roll-up. Instead, each specialist is scored only over the repos matching its language, and Vulkro is re-scored over the same repo subset. Every comparison below is a fair fight. Vulkro wins two of the three; the one it loses is printed just as large.

Bandit vs Vulkro, 7 Python repos (42 catalogued bugs)​

Bandit 1.9.4Vulkro (same repos)
Precision0.560.87
Recall0.430.64
TP / FP / FN18 / 14 / 2427 / 4 / 15

On its home language, Bandit finds 18 of the 42 catalogued Python bugs with 14 false positives; Vulkro finds 27 with 4.

Gosec vs Vulkro, 1 Go repo (4 catalogued bugs): Gosec wins​

Gosec 2.27.1Vulkro (same repo)
Precision0.200.00
Recall0.750.00
TP / FP / FN3 / 12 / 10 / 1 / 4

On our one Go repo, Gosec finds 3 of 4 bugs; Vulkro finds 0. Go detection depth is a known gap we publish. The four misses (a fmt.Sprintf SQL interpolation, an html/template XSS, and two HttpOnly: false insecure cookies in govwa) stay catalogued as false negatives rather than being quietly dropped from the corpus. If Go is your primary language, run Gosec; it is free and it beats us there today.

njsscan vs Vulkro, 4 JS/TS repos (30 catalogued bugs)​

njsscan 0.4.3Vulkro (same repos)
Precision0.500.77
Recall0.330.57
TP / FP / FN10 / 10 / 2017 / 5 / 13

One caveat that the per-repo rows keep visible: njsscan targets plain JavaScript, so on the TypeScript repo (juice-shop) it sees almost nothing in .ts files and scores 0 of 8 catalogued bugs there.

Max-recall mode​

Drop the confidence threshold (--min-confidence low, the default) and Vulkro trades precision for recall. The published scorecard labels this run as max-recall:

metricVulkro max-recallVulkro headline (high)
Precision0.150.81
Recall0.720.58
TP / FP / FN55 / 314 / 2144 / 10 / 32

Max-recall finds 55 of the 76 roll-up bugs at the cost of 314 false positives. It is the right setting for a one-shot audit where a human will triage everything once; --min-confidence high is the right setting for CI. We publish both so the trade-off is yours to make, not ours to hide.

Where Vulkro misses​

The corpus is built to keep our failures on the record:

  • GraphQL resolver taint. DVGA (Damn Vulnerable GraphQL Application) has 6 catalogued bugs reached through GraphQL resolvers. Vulkro finds 0 of them. Tracing taint through resolver dispatch is an open detection gap.
  • Go. govwa has 4 catalogued bugs; Vulkro finds 0, Gosec finds 3 (see the fair-fight table above).
  • PHP and Java. DVWA (7 bugs, PHP) is outside Vulkro's language scope entirely. WebGoat (4) and java-sec-code (3) are Java: this scorecard was generated with Vulkro 0.15.0, before Java/Spring support shipped, so those repos score 0 here and stay excluded from the roll-up until the corpus is re-scored with Java analysis on. Bearer finds 9 of those 14 catalogued bugs. Java coverage today is documented on the Java and Spring page.

By bug class, the 32 headline-run misses across the supported-language repos concentrate in: SQL injection (11), command injection (8), XSS (6), open redirect (4), plus single-digit counts of insecure deserialization, insecure cookies, SSRF, JWT verification, path traversal, and LFI. The full FN-by-class table is regenerated in every scorecard.

Speed​

Wall clock for the whole 15-repo corpus, one full end-to-end process invocation per tool per repo. Measured 2026-07-04 on an Apple M5 Pro, 24 GB RAM, macOS 26.5.1.

toolwhole-corpus scan time
Gosec 2.27.10.6s (1 Go repo only)
Bandit 1.9.43.2s (7 Python repos only)
njsscan 0.4.324.1s (4 JS/TS repos only)
Vulkro 0.15.0 (high confidence)38.0s
Vulkro 0.15.0 (default)39.3s
Opengrep 1.25.046.1s
Semgrep CE 1.136.048.9s
Bearer 2.0.2363.0s

The honesty notes that come with that table:

  • This reference is a single cold run per tool per repo (--timing-runs 1), because the 7-tool matrix makes median-of-3 a very long session. For publishable single-tool comparisons the harness supports --timing-runs 3 and we recommend it; the aggregation line embedded in the scorecard records exactly what each run did.
  • Repos are pre-cloned to local disk before timing; clone time is excluded. Every sample includes process start, scan, and report serialization.
  • Tool-internal result caches are disabled where the tool supports it (Bearer runs with --force), so every sample is a cold scan. Bearer's incremental cache makes its repeat scans on unchanged code much faster than shown here; the table just does not mix cold and warm numbers.
  • The specialists only scan their matching-language repos, so their totals cover only that subset. A 0.6s Gosec total is not comparable to a 38.0s whole-corpus total and the table says so.
  • Wall clock is machine-dependent. Compare the ratios between tools on the same machine, not absolute seconds across machines.

Reproduce it yourself​

You do not have to trust any number on this page. The zero-to-scorecard walkthrough for a fresh machine is bench/comparison/REPRODUCE.md; the short version:

# 1. Prerequisites: the vulkro binary, git, python. Competitor tools
#    are optional; without them you get a vulkro-only scorecard.

# 2. The one command (clones the pinned corpus, scans, scores, renders):
VULKRO="$(command -v vulkro)" bench/comparison/run.sh --tier1

# 3. The published headline setting:
VULKRO="$(command -v vulkro)" bench/comparison/run.sh --tier1 --min-confidence high

# 4. Publishable speed numbers (median-of-3):
VULKRO="$(command -v vulkro)" bench/comparison/run.sh --tier1 --timing-runs 3

# 5. Read the result:
cat bench/comparison/results-*/scorecard.md

Each run writes raw findings, normalized JSON, the markdown scorecard, runtime.json, and timing_meta.json (hardware, run count, cache state, and the tool versions actually used).

What "reproducible" means here: same Vulkro binary + same corpus SHAs

  • same scoring code = same TP/FP/FN counts, every run, on any machine. Known sources of drift, documented in REPRODUCE.md:
  • Semgrep rules are not pinned. The p/security-audit ruleset is served live by the Semgrep registry and changes over time, so Semgrep and Opengrep counts can move without any change on our side. Our corpus SHAs and scoring code are pinned; a competitor's rule feed is not ours to pin.
  • Competitor versions. Newer Bearer or specialist releases change rules. The scorecard records what you ran; compare like with like.
  • Hardware moves speed numbers. Compare ratios, not absolute seconds.

If you get a different Vulkro TP/FP/FN on a pinned corpus, that is a bug: report it with your results-<timestamp>/ directory attached.

Tool versions in the published run​

toolversionnote
Vulkro0.15.0scored at --min-confidence high (headline) and low (max-recall)
Semgrep CE1.136.0p/security-audit ruleset, fetched live, not pinned upstream
Opengrep1.25.0same live p/security-audit ruleset as the Semgrep lane
Bearer2.0.2--scanner sast lane, --force (cache disabled)
Bandit1.9.4Python repos only (specialists section)
Gosec2.27.1Go repos only (specialists section)
njsscan0.4.3JS/TS repos only; bundles its own semgrep 1.86.0 engine

Tools not included​

Snyk Code is deliberately not part of the published runs: it requires an authenticated account (snyk auth), and publishing benchmark results may be restricted by its Terms of Service (a check we have not completed). The harness keeps a fully working snyk runner: add snyk to --tools and authenticate to produce the numbers locally for yourself, after checking the current Snyk ToS. The scorecards carry a standing "Tools not included" note so this gap is a documented decision, not a silent one.

Akto and Pynt are DAST tools; they cannot scan a static code directory at all and exist only as documented stub lanes in the harness.

Caveats: what these numbers do not tell you​

  • The corpus is deliberately-vulnerable code. Real-world performance varies; deliberately-vulnerable apps are denser in bugs and lighter in framework noise than production code. The 3 clean framework controls (express, fastapi, flask) are the noise-floor check inside the corpus.
  • Class matching matters. A finding under the wrong class does not count as a TP even when the line is right. This penalizes every tool roughly equally.
  • Ground truth is a floor, not a census. The 90 catalogued bugs are the bugs we could verify by provenance; the OOS bucket exists precisely because we do not pretend the rest of every file is clean.
  • A benchmark is one input. Run the tools on your own code. The harness exists so that "run it yourself" is a one-command suggestion rather than a slogan.