2026-04-11 Triage Ablation Results

Published 2026-04-11. Updated 2026-04-12 with batch-2 reruns after disabling EGATS in moat aliases. This page is kept as an archival experiment log with full numbers and raw links.

Executive summary

On 2026-04-06, 0 v0.6.0 shipped with an 11-layer triage stack for false-positive reduction (internally named “the moat”). On 2026-04-11, we ran a 21-profile ablation across XBOW white-box, XBOW black-box, npm-bench, and stubborn-14 to measure each layer’s net effect.

• XBOW white-box: two fewer flags versus none, 1.6× cost per flag, and 63% fewer findings.
• XBOW black-box: more flags, lower cost per flag, and 48% fewer findings.
• On npm-bench, the 11-layer moat is a no-op in batch 1: default and moat produce identical numbers. Batch-2 reruns indicate large variance at this sample size and require repeated runs for stable FPR conclusions.
• One specific layer — egats, our tree-search module — regresses performance on the hardest slice and was removed from default moat aliases.

System under test

The study exercised 0’s target discovery, shell probes, and structured findings. Shell execution used the host. The evaluation suites were:

• XBOW validation benchmarks — 104 web challenges with real flags. White-box gives the agent source code, black-box gives it only the running service.
• npm-bench — our own 81-package corpus: 27 known-malicious packages, 27 known-vulnerable packages, 27 known-safe packages.
• stubborn-14 — the 14 XBOW challenges that 0 has historically failed on. A worst-case slice.

The none profile solved 86% of the first 50 XBOW white-box challenges.

As of 2026-05-06, the retained-artifact aggregate was 103/104 (99.0%), with white-box at 102/104 (98.1%). The gpt-5.4 black-box cohort was 93/95 (97.9%); aggregate retention changes with GitHub’s 90-day artifact window. Cybench scored 36/40 (90.0%), single-config and single-shot. Reported external XBOW figures were Shannon 96%, KinoSec 92%, BoxPwnr 97.1%, and Cyber-AutoAgent 84%, under different protocols.

Triage stack under test

The moat is 11 triage layers. A finding from the attack agent is passed through them in sequence, and any layer can reject, downgrade, or enrich the finding:

Each layer has prior-art lineage. adversarial_debate follows Anthropic/Irving debate papers. egats_tree_search is an adaptation of MAPTA-style evidence-gated tree search. pov_gate mirrors proof-of-vulnerability gating ideas similar to Endor Labs reachability positioning. evidence_gate and structured_verify are closest to Semgrep Assistant-style LLM post-filtering.

The experiment tested the earlier expectation that successive layers would reduce FPR below 5%.

Ablation methodology

An ablation study takes a multi-component system and disables components one at a time (or in groups) to estimate each component’s marginal contribution. The key requirement is a representative evaluation slice; otherwise the result measures slice bias more than system behavior.

Prior evaluations and limitations

Before today, we had two internal data points on the moat:

Data point one: On a 30-package slice of npm-bench, 0 scored F1=0.444 with the default profile and we logged a recall concern. On the current 81-package slice, every profile shows TPR=1.00. The earlier 30-package result was superseded and no longer represented the live set.

Data point two: A stubborn-14 ablation showed regression from 4 flags to 0. This was useful for diagnosing failure modes, but stubborn-14 is a worst-case slice by construction and is not representative for product-level ship decisions.

Use representative slices for release decisions and stubborn slices to diagnose failures.

The ablation matrix

Today’s run is 21 configurations across three benchmark families:

Profile definitions, for readers who want to reproduce:

• none — attack agent only. No triage, no stable features, no early-stop, no script templates, no progress handoff.
• no-triage — stable features on, moat off. This is the “what does our engineering work do without triage” baseline.
• moat-only — moat on, stable features off. The inverse of no-triage.
• moat — moat on, stable features on.
• default — everything on. Our shipping configuration at the time of the run.

Result 1: XBOW white-box — the moat costs us flags

Profile	Flags	Solve rate	Findings	Cost	$/flag
none	43/50	86%	67	$14.34	$0.33
no-triage	44/50	88%	67	$17.17	$0.39
moat-only	41/50	82%	25	$26.89	$0.66
moat	41/50	82%	25	$21.82	$0.53

Relative to none, moat lost two flags (43 → 41), reduced findings 63% (67 → 25), and raised cost per flag 1.6× ($0.33 → $0.53).

Result 2: XBOW black-box — the moat strictly dominates

Profile	Flags	Solve rate	Findings	Cost	$/flag
none	18/25	72%	27	$13.72	$0.76
no-triage	19/25	76%	34	$10.37	$0.55
moat-only	18/25	72%	13	$11.22	$0.62
moat	19/25	76%	14	$10.04	$0.53

moat improved all three reported black-box measures: 19 versus 18 flags, 14 versus 27 findings (48% fewer), and $0.53 versus $0.76 per flag. One hypothesis is that source access produces stronger candidates before triage; this experiment does not isolate that cause.

Result 3: npm-bench — the moat is a literal no-op

Profile	F1	TPR	FPR	Safe correct
none	0.973	1.00	0.11	24/27
no-triage	0.964	1.00	0.15	23/27
moat-only	0.964	1.00	0.15	23/27
moat	0.956	1.00	0.19	22/27
default	0.956	1.00	0.19	22/27

moat and default matched: F1=0.956, TPR=1.00, FPR=0.19, and 22/27 safe packages correctly classified. Across profiles, FPR was 0.11 → 0.15 → 0.15 → 0.19 → 0.19. Batch 1 suggested an interaction with stable features; batch 2 weakened that attribution. Repeated runs are needed.

Result 4: single-feature isolation on stubborn-14

To figure out whether any specific moat layer was pulling its weight, we ran seven single-feature profiles against stubborn-14, each adding exactly one moat layer on top of default-ref:

Profile	Flags	Delta	Cost	$/flag
default-ref	2/14	—	$7.24	$3.62
+pov	4/14	+2	$9.56	$2.39
+reach	5/14	+3	$8.04	$1.61
+multi	3/14	+1	$7.55	$2.52
+debate	5/14	+3	$13.26	$2.65
+mem	4/14	+2	$13.40	$3.35
+egats	1/14	−1	$15.93	$15.93
+cons	3/14	+1	$8.01	$2.67

Six of seven layers are net-neutral-to-positive on stubborn-14 when measured individually. One layer regresses: egats. It goes 2 → 1 flags, for 10× the worst per-flag cost in the table, on a slice where every other layer produces new flags.

This likely explains the earlier severe regression result. When the full moat runs, egats appears to prune exploration branches that other layers would use to find flags, producing net-negative interactions on the hardest slice. With egats disabled, moat profiles are net-positive across measured slices.

We filed 0#116 to disable EGATS in default aliases. The code remains available but opt-in. A separate postmortem is planned to document where the implementation diverges from MAPTA-style scoring.

Which layer helps on which slice

Putting the cross-slice picture in one place:

There is no single profile that wins on all three benchmark families. no-triage wins XBOW white-box by raw flag count. moat wins XBOW black-box in strict Pareto. none wins npm-bench on FPR. The right triage policy is slice-dependent, which means any static shipping profile is a compromise.

Methodology takeaways

• Evaluate representative and difficult slices separately.
• Retain benchmark versions: the earlier F1=0.444 result used 30 packages; the 81-package set reached TPR=1.00.
• Report black-box and white-box results separately.
• Repeat runs before attributing FPR changes to a subsystem.
• Isolate features when aggregate behavior is surprising. The seven-profile stubborn-14 run took about six runner-hours and identified EGATS regression.

Implementation updates

Three things are in flight as of this writing:

Shipped today: per-finding layer verdicts telemetry (commit 6f1a889, closes 0#112). Every finding now logs which triage layer touched it, what verdict each layer returned, how long it took, and how much it cost. This is the training signal we need for the next thing.

Shipped today: egats excluded from the moat and moat-only profile aliases in CI (0#116). The implementation needs a rewrite against the original MAPTA scoring function before it goes back in.

Shipped today: triage-dataset-v1.jsonl — 969 labeled rows from the 21 ablation runs. Each row carries the finding text, the 45-element handcrafted feature vector, per-layer telemetry where available, and the ground-truth label. This is the first training-data artifact for the learned routing model below.

Next sprint: learned dynamic routing (0#113). Because no static policy wins on all three slices, we’re going to train a small classifier that picks which triage layers to run per-finding based on finding metadata (class, confidence, evidence type) and benchmark mode (white-box/black-box/package-scan). The per-finding telemetry is the training data. The architecture is inspired by VulnBERT (Guanni Qu, Pebblebed Research Residency) — a hybrid CodeBERT + 51 handcrafted features classifier for Linux kernel vulnerabilities that hits 91.4% recall at 5.9% FPR. We’re exploring whether the same hybrid approach applies to web vulnerability findings.

The npm-bench stable-features FPR regression is a separate workstream. We have a suspect (the script-template library and progress-handoff injection are too aggressive on generic package analysis) and we’ll publish a followup when we have numbers.

Conclusions

Effects varied by slice and batch. EGATS was removed from default aliases; telemetry and profile controls supported further measurement. Learned routing remained follow-on work. The aggregate below-5%-FPR expectation was unsupported.

Acknowledgments and prior art

This work builds on prior published systems and methods.

• VulnBERT (Guanni Qu, Pebblebed Research Residency) — the hybrid CodeBERT + 51 handcrafted-feature classifier whose recall/FPR numbers are the bar we’re trying to clear on the learned-routing side. The same hybrid feature-engineering methodology (handcrafted features + neural embeddings + cross-attention fusion) is what we’re adapting for web vulnerability findings.
• MAPTA (arXiv:2508.20816) — the evidence-gated tree search technique that inspired our EGATS layer.
• Anthropic’s Debate (arXiv:2402.06782) — the lineage for our adversarial_debate layer. Per the single-feature isolation, debate contributes +3 flags on stubborn-14, which is tied for best.
• All You Need Is A Fuzzing Brain (arXiv:2509.07225) informed the bounded PoC-generation gate.
• Endor Labs — their reachability framing influenced our reachability gate design and precision target framing.
• Semgrep Assistant — the 96% auto-triage number that is the other bar in our release notes. Their LLM post-filter architecture is very close to our evidence_gate + structured_verify combination.
• BoxPwnr (0ca) — context compaction, loop detection, and progress handoff patterns informed corresponding 0 features.

Report methodology errors through the linked tracking issues.

Links and references

• FP Reduction Moat — the design doc for the 11-layer moat, now rewritten with the measured numbers from this ablation
• 0#72 — the ablation matrix issue, with run IDs and per-comment result tables
• 0#111 — the npm-bench “recall problem” that turned out not to exist on the live test set (closed)
• 0#112 — per-finding layer verdicts telemetry (closed by commit 6f1a889)
• 0#113 — learned dynamic routing for triage (open)
• 0#114 — triage-dataset-v1.jsonl generation (closed by commit f40e1c1)
• 0#116 — disable egats in default profile (closed by commit aadcf32)
• triage-dataset-v1.jsonl — 969 labeled rows, the first training-data artifact

Follow-up: batch 2 results (2026-04-12)

We re-ran the full limit=50 white-box matrix against the post-EGATS-disable commit (aadcf32). The moat and moat-only profiles now run without egatsTreeSearch.

White-box @ limit=50 — before and after egats removal

Profile	Batch 1 (with egats)	Batch 2 (without egats)	Δ flags	Δ cost
none	43/50	44/50	+1	+$2.05
no-triage	44/50	43/50	−1	+$4.90
moat-only	41/50	42/50	+1	−$10.94
moat	41/50	42/50	+1	−$5.36

Removing egats improved the moat by 1 flag and dropped cost by 25%. All four profiles are now within 2 flags of each other (42–44). The moat-vs-baseline gap went from 3 flags (batch 1) to 1–2 flags (batch 2) — well within LLM noise at N=50.

Combined batch interpretation: with EGATS disabled, the moat costs at most 1-2 flags on white-box for ~60% fewer findings at roughly similar cost.

npm-bench batch 2 — the noise finding

Batch 1 showed default at FPR 0.19 vs none at 0.11, which we interpreted as “stable features cause the FPR increase.” Batch 2’s default run got FPR 0.11 — matching none. The 0.19 from batch 1 was probably a 2-package noise swing on 27 safe packages.

We also ran single-feature isolation:

Profile	F1	TPR	FPR	Key finding
default (v2)	0.973	1.00	0.11	Matches batch 1 none
no-script-templates	0.964	0.98	0.11	Loses 1 detection — templates help recall
no-handoff	0.973	1.00	0.11	No effect
no-early-stop	(timed out)

Batch 1’s “stable features cause FPR” claim is likely noise. The same-profile FPR swings ±0.08 between runs. At N=27 safe packages, that’s a 2-package flip — within expected LLM variance. We’d need repeat=3+ per profile to separate signal from noise on npm-bench FPR.

Script templates help recall on this run: disabling them loses one detection (TPR 1.00 → 0.98).

What held across both batches

These findings replicated:

• egats is the broken layer — removing it improved moat by +1 flag and 25% cost drop
• The moat cuts findings by ~60% consistently (67→25 batch 1, 72→27 batch 2)
• 100% TPR on npm-bench across every profile and both batches
• Black-box moat is strong (37/50 at limit=50 on moat-only, consistent with batch 1’s 36/50)
• Per-finding layerVerdicts telemetry works — the v2 dataset runs show 8/14 findings with populated verdict arrays

What didn’t replicate

• “Stable features cause the npm-bench FPR increase” — batch 2 default matches batch 1 none (FPR 0.11 both). The 0.19 was probably noise.
• “The moat costs 2 flags on white-box” — after removing egats, the gap is 1-2 flags and within expected run noise.

Combined interpretation

With EGATS disabled, the measured white-box finding reduction was about 60% at a 0–2 flag cost. Black-box results improved. npm-bench FPR attribution needs repeated runs because the observed differences were within run-to-run variation.

Current status

• The measured layer behavior from this run is integrated into FP Reduction Moat.
• The benchmark-level interpretation lives on Benchmark.
• The follow-on routing work is tracked in Dynamic Routing Design and 0#113.