Dynamic Triage Routing — Design Doc

Historical design and April 2026 results, tracked in 0#113. See rule-based v0 for the implementation record. The original 45-feature proposal below predates the current 55-feature extractor. Keep feature revisions aligned when reusing its datasets. Current hand-coded TP/FP scoring and rule-based layer selection are separate, default-off mechanisms; the trained artifacts do not imply a deployed learned layer-selector or optimal routing among LLM providers.

Motivation

The 2026-04-11 ablation (writeup, data) found different best policies by slice: no-triage for white-box flag count, moat for black-box, and none for npm FPR. On stubborn-14, reachability added three flags at $1.61/flag; EGATS lost one at $15.93/flag. The proposed router selects layers per finding.

Design goals

1. Select layers per finding while preserving operator feature controls.
2. Match or exceed the best static profile on every slice.
3. Keep inference below one millisecond using local features and CPU execution.
4. Expose the features behind each routing decision.
5. A/B test behind a default-off flag before changing defaults.

What the router sees

Two classes of inputs:

Per-finding inputs (from the attack agent output):

• The 45-element handcrafted feature vector from feature-extractor.ts — same vector VulnBERT-style hybrid architectures consume
• Encoded category (sql-injection, xss, ssrf, information-disclosure, etc. — one-hot over ~20 categories)
• Severity ordinal
• Agent-assigned confidence (0.0–1.0)
• Presence of CWE/CVE references
• Evidence completeness score (derived from the feature vector)

Per-scan inputs (constant across a scan):

• Mode (white-box / black-box / mcp / web)
• Target type (web-app / url / npm-package / source-code / oci-image)
• Benchmark slice when applicable (xbow / npm-bench / production / unknown)

Mode and target type may help predict useful layers. Their value requires held-out evaluation.

What the router outputs

A multi-label verdict over the 10 triage layers (the 6 triage-stage layers covered by 0#112’s telemetry plus the 4 verify-stage layers that ship to telemetry in v2):

interface RouterOutput {
  // Primary output: which layers should run for this finding
  runLayers: TriageLayerName[];
  skipLayers: TriageLayerName[];

  // TP/FP probability — same head as a standalone triage classifier
  // would use. Bypasses the layers entirely when confidence is extreme.
  tpProbability: number;

  // Decision shortcuts
  autoAccept: boolean;  // tpProbability > accept_threshold → skip all layers, mark accepted
  autoReject: boolean;  // tpProbability < reject_threshold → skip all layers, mark FP

  // For debugging / interpretability
  reason: string;       // short human-readable explanation
  featureImportances?: Record<string, number>;
}

autoAccept and autoReject can bypass further layers when their acceptance criteria hold. runLayers selects checks for remaining findings. Measure the recall cost of either shortcut.

Training signal

Every layer verdict entry logged by 0#112 is a training example. A finding that accumulates layerVerdicts of the form:

[
  { "layer": "holding_it_wrong", "verdict": "pass", "durationMs": 0.3, "costUsd": 0 },
  { "layer": "evidence_gate", "verdict": "pass", "confidence": 0.83, "durationMs": 0.1, "costUsd": 0 },
  { "layer": "oracle", "verdict": "downgrade", "confidence": 0.4, "reason": "only 1/3 sqli signals fired", "durationMs": 4231, "costUsd": 0 }
]

is a labeled example of “which layers mattered for this finding.” The router is trained to predict, for each layer independently:

1. Would the layer change the final verdict if we ran it? If the answer is “no, it would just pass or be skipped,” the router learns to skip it. If “yes, it would reject or downgrade,” the router learns to run it.
2. Was the layer worth the cost? layerVerdicts[i].costUsd and layerVerdicts[i].durationMs let us compute a “cost saved per verdict change” ratio per layer type. Layers with near-zero cost (holding_it_wrong, evidence_gate, reachability, oracles — all regex/grep/deterministic) should essentially always run. Layers with real LLM cost (structured_verify, consensus, adversarial_debate) should only run when the router predicts they’re likely to flip the verdict.

The ground-truth final verdict comes from:

• Flag extraction for XBOW rows (flag found = true positive)
• Package verdict for npm-bench rows (malicious/vulnerable = true positive; safe = false positive)
• Blind verify status for local scan DB rows

See the Triage Dataset page for the full JSONL schema and 0#114 for triage-dataset-v1.jsonl (969 rows from the 21 ablation runs, with layer_verdicts populated on rows from commits post-6f1a889).

Model class

Three candidates, in order of preference:

Option A: XGBoost multi-label head on the 45-feature vector

• CPU inference below one millisecond, feature importance, and a small distributable model.
• Uses the 45 handcrafted features; finding text is excluded.
• Evaluate first on the available 1k–10k-row dataset. VulnBERT’s feature-only ablation reported 76.8% recall on kernel commits.

Option B: Small MLP head on fused (features, CodeBERT embedding)

• Combines features with microsoft/codebert-base, following VulnBERT’s cross-attention design (92.2% recall / 1.2% FPR on kernel commits).
• Approximately 125M parameters; estimated inference is ~10 ms on CPU or ~1 ms on GPU. Training and model distribution add requirements.
• Consider if measured accuracy gains justify the additional inference cost.

Option C: Knowledge distillation from a larger LLM

• Distill a larger text model, such as GPT-5.4-mini, into a routing student.
• Requires a more expensive training and distribution pipeline.
• Consider after evaluating options A and B.

Evaluate XGBoost first. Use its measured errors and cost to decide whether a neural model is justified.

Phase 3 results (2026-04-12): Option A trained and evaluated

XGBoost (100 trees, depth 5, focal-loss-style scale_pos_weight) trained on triage-dataset-v2.jsonl (1514 rows). Model at packages/benchmark/results/triage-router-v1.json.

Aggregate 5-fold CV: F1=0.944, precision=0.969, recall=0.920, AUC=0.886.

Leave-one-slice-out (the generalization test):

Held-out	F1	TP recall	FP recall
npm-bench	0.664	50%	84%
xbow-bb	0.859	78%	33%
xbow-wb	0.900	93%	12%

Cross-slice generalization is poor. A model trained on xbow catches only 50% of npm-bench TPs. Adding slice-type indicators (+3 features) improved npm-bench to 0.705 — marginal.

Per-slice classifiers (Path B) are the clear winner:

Slice	Within-slice F1
npm-bench	0.930 ± 0.025
xbow-wb	0.914 ± 0.023
xbow-bb	0.721 ± 0.363 (n=115)

Feature importance is completely different per slice:

• npm-bench: text_description_length (50%) — longer descriptions predict TP
• xbow-bb: cross_response_request_length_ratio (53%) — bigger response ratio predicts TP
• xbow-wb: req_path_traversal (12%), req_param_count (12%), resp_error_message (7%) — actual exploit indicators

Decision: Per-slice classifiers (Path B) are the deployment target. The scanner knows its mode + target-type at scan start — dispatch to the right classifier. Each runs sub-millisecond on CPU. The augmented single-model approach (Path A) does not clear the bar for npm-bench generalization.

Deployment options (ordered by shipping speed):

1. Hand-coded thresholds from the model’s learned splits — ships today in TypeScript, zero deps
2. JS XGBoost loader — npm package that reads the JSON model, moderate accuracy
3. ONNX runtime — export to ONNX, load via onnxruntime-node, highest accuracy, adds native dep

Training objective

For the multi-label routing head (one binary classifier per layer):

$$\mathcal{L}{\text{route}} = \sum{l \in \text{layers}} w_l \cdot \text{BCE}(\hat{y}_l, y_l)$$

where $y_l = 1$ if the layer changed the final verdict on this finding’s ground truth, $0$ otherwise, and $w_l$ is a per-layer weight that penalizes false negatives more on expensive layers (we’d rather accidentally run a cheap layer than accidentally skip an expensive one when it would have caught a FP).

For the TP/FP head (binary classification):

$$\mathcal{L}_{\text{tp}} = \alpha \cdot \text{BCE}(\hat{p}, y)$$

where $y \in {0, 1}$ is the final verdict (flag found / package verdict / blind verify status) and $\alpha$ balances the head against the routing loss.

Combined: $\mathcal{L} = \mathcal{L}{\text{route}} + \lambda \cdot \mathcal{L}{\text{tp}}$, with $\lambda$ swept over ${0.1, 0.5, 1.0, 2.0}$ in the ablation.

The v1 dataset contains 884 TP and 85 FP rows (91.2% TP). Evaluate focal loss for this imbalance.

Evaluation

Three metrics, one per benchmark slice:

1. XBOW white-box flag count at limit=50. Bar to clear: no-triage (44/50). The router should pick layer subsets that match or exceed this on white-box findings, while still getting 63% finding-count reduction (matching moat on that axis).
2. XBOW black-box flag count at limit=50. Bar to clear: moat (19/25 at limit=25, extrapolates to ~38/50). The router should match or exceed.
3. npm-bench F1 on the 81-package set. Bar to clear: none (F1 0.973, FPR 0.11). The router should match or exceed.

Plus a cost metric: dollars spent per flag on each slice. A router that matches flag count but costs 2× is not shipping.

And a recall metric: per-category recall breakdown. No category should lose more than 5% recall vs the per-slice baseline. If a router fails, say, SQLi findings badly while winning on aggregate, that’s a ship blocker.

Rollout plan

1. Phase 1: design doc (this page). Gather feedback on 0#113. Target: ~1 week.
2. Phase 2: training data v2. Re-run the 21-profile ablation matrix against a commit that has 0#112’s layerVerdicts populated across the board. This produces triage-dataset-v2.jsonl with per-layer supervision on every row. Target: ~3 days.
3. Phase 3: Option A XGBoost baseline. Train, evaluate against the three bars above, report results publicly. Target: ~1 week.
4. Phase 4: decision. If Option A clears the bar, ship it behind ZERO_FEATURE_LEARNED_ROUTER=1, A/B test in CI, promote to default when stable. If Option A plateaus, proceed to Phase 5.
5. Phase 5 (contingent): Option B cross-attention model. Fine-tune CodeBERT + feature projection + routing head on v2 dataset. Target: ~3-4 weeks (requires GPU, distribution pipeline, inference integration). This is the option most aligned with the VulnBERT hybrid architecture.
6. Phase 6: paper. Submit to a security venue (IEEE S&P, USENIX Security, CCS). Scope: “Learned dynamic triage routing for LLM-agent vulnerability scanners.” First half of the empirical section is the 2026-04-11 ablation results log; second half is the router’s measured improvement over the best static profile per slice.

Open questions

• How do we handle the imbalance on the v1 dataset? 91.2% TP is way outside the range the VulnBERT paper’s focal-loss tuning was validated on. We may need to undersample TP rows during training, or move to a more TP/FP-balanced dataset (possibly by running 0 against more benign npm packages to generate more FPs).
• Does the router get to see the attack agent’s conversation history? Right now the 45 features only see the finding itself. The agent’s reasoning trail (which strategies it tried, what signals it followed, what it gave up on) might contain useful signal for the router. But including it risks making the router effectively a full LLM call, which defeats the sub-millisecond goal.
• Can the router bypass layers the agent already implicitly covered? For example, if the attack agent successfully exploited an SQLi and captured a flag, the oracle layer would re-attempt the exploit and presumably succeed. Running it is redundant. The router could learn “if the finding has a flag-shaped response, skip all oracles.”
• How do we avoid benchmark overfitting? XBOW is 104 challenges. npm-bench is 81 packages. If the router learns the specific finding shapes of these two benchmarks, it won’t generalize to production targets. We need a held-out slice that isn’t in any training set — possibly a fresh sweep against production bug bounty programs with manual ground-truth labeling.
• What’s the right interaction with 0#116 (egats disable)? Should the router learn when egats would help (and opt it in for those findings)? Or should we treat the layer as permanently off until we rewrite it against the MAPTA scoring function? Currently leaning toward the latter — a broken layer shouldn’t be in the router’s action space.

Related

• 2026-04-11 Ablation writeup — the measurement that motivates this design
• FP Reduction Moat — the static triage stack this design replaces
• Finding Triage ML — the original hybrid ML design doc, this page supersedes its routing section
• Triage Dataset — the JSONL schema the router trains on
• Feature Extractor — the current 55-feature reference (45 in the original proposal)
• 0#72 — the ablation data, with run IDs and per-comment result tables
• 0#112 — per-finding layerVerdicts telemetry (prerequisite, shipped 2026-04-11)
• 0#113 — this tracking issue
• 0#114 — triage-dataset-v1.jsonl (first training data, shipped 2026-04-11)
• 0#116 — disable egatsTreeSearch by default (done 2026-04-11)
• VulnBERT — Guanni Qu, Pebblebed Research Residency — the hybrid classifier architecture this design is modeled on (91.4% recall / 5.9% FPR on Linux kernel commits)