Skip to content

Appendix D: the verifier study#

Study scope#

This is a completed, pre-registered two-arm study. It compared the same system with the verifier enabled and disabled against a disposable lab target, and it measured how verification changed the findings included in the final report. The verifier is the control chapter 03 and chapter 06 describe: the one thing in this system that can accept or refuse a raised severity. The sections below give the design, the numbers it produced, everything that went wrong while running it, and exactly what it still does not prove. Appendix E is the completed follow-up that separates the model verifier from the deterministic handling of its verdicts. The scheduler and ranking layers are a different question, which this study did not touch; their status, with every other study's, is in the evidence register.

Study design#

Two configurations of the same system, run against the same public lab target, ten fresh runs each, 20 runs total. What gets ablated is a package, not a single rule: a model verifier plus the deterministic handling of its verdict, the accept-or-refuse-or-raise logic sitting downstream of whatever the model decides. This study does not isolate that deterministic handling from the model verifier feeding it; turning the pair off together is the only arm run. One configuration runs the pair as shipped; call it FULL. The other has that stage turned off; call it NOVERIFY. Turning off that stage left the write-time governor in place. This is the committed suppression note's name for the deterministic check that fires when a finding is stored, upstream of the ablated verify-and-accept stage. It operated in both arms; the suppression counts show it catching a finding in two NOVERIFY runs. The precise claim is that NOVERIFY removes the model verifier and the deterministic verdict handling behind it, so once a finding is stored, nothing reviews it again before the report; it is not a raw agent with every check removed. The order the 20 runs actually happened in was fixed by a coin-flip-equivalent shuffle before the first run started, not chosen as they went. Every run started from a freshly rebuilt copy of the target and a clean state directory (no memory of any earlier run, in either arm) so one run's luck could not leak into the next. The instructions handed to the two arms were byte-identical except for one mechanical sentence telling NOVERIFY that the verify step was switched off; neither arm's instructions used the words study, experiment, arm, or measure, or hinted at what a good result would look like. And an undeclared decoy host sat on the network the whole time, checked for contact before and after every single run, so any scan wandering off the one authorized target would have left a mark somewhere neither arm had been told to look.

The four things decided before run one, in order of how much they matter: what counts as suppressed, what counts as a shipped finding, what the exclusion rule is, and what statistical test would be run and at what threshold. None of those moved once the runs started. The one exclusion rule that existed (infrastructure dead at launch) never had to fire; every run that started, finished, and every one of the 20 is in the numbers below.

Results summary#

Both configurations repeatedly found several of the same core bugs, with substantial per-run variation: an authentication bypass through injection appeared in every run's findings; a full-table dump through a UNION-based injection and an object-reference flaw appeared in most runs; a mass-assignment path to a privileged role shipped in 6/10 of the FULL runs and 3/10 of the NOVERIFY runs. The verifier does not decide what a model can find. What moved between the two arms was what actually reached the report afterward: the noise the model itself proposed and then, in one arm, had rejected before anyone read it.

Measured outcomes#

Findings marked false before the report went out (the thing the verifier exists to catch) had a median of 2.0 per run with the verifier on, against 0.0 with it off (exact one-sided Mann-Whitney U, 0.00003: the chance of a gap that size under pure noise). A fresh model, shown the shipped findings with every arm label stripped off, scored FULL's precision at a median of 0.471 against NOVERIFY's 0.353 (0.00869), supported, not confirmed, on this model-blinded pass. A straight shipped-finding count was also pre-registered and did not clear this study's significance bar. The fourth pre-registered measure, a check for criticals shipped below their deserved band, carried no predicted direction but did carry a pre-specified two-sided test: 17 such events under FULL against 15 under NOVERIFY (exact two-sided Mann-Whitney U, as pre-registered, 0.854). No detected difference, and a pre-registered measure that comes up empty still gets reported as one, not left out.

Three honesty notes belong beside those numbers, not after them. First, distributions overlap at edges; claim is the tested median difference: the lowest precision score in FULL still landed below the highest score in NOVERIFY, so no single cutoff separates the two arms, and the number above is a median gap, not a clean split. Second, the adjudicator that produced the precision numbers was itself a model, of the same family running the scans it was grading. A human doing the same blind pass over the same stripped packet has not happened yet (this book's own numbers record that plainly, as model_blinded_human_pass_pending) and that pass, not this one, is the gate this book asks for before the precision result moves from supported to confirmed; call it supported, model-blinded, publication-grade confirmation pending, and nothing stronger until that pass exists.

Third: recall against a curated ground-truth list carried no pre-registered direction, so it is reported two-sided, not one-sided. No statistically significant recall difference was observed (0.158); this study was not designed to establish equivalent recall, no non-inferiority margin was set, and the honest reading is silence, not a clean bill of health for turning the verifier off. The ground-truth list itself double-counts one condition: the login SQL injection appears twice, as the auth_bypass entry juice-auth-001 and the sqli entry juice-sqli-002, both at the same login endpoint, and the committed per-run match lists carry both entries in every run. Deduplicating the pair to one condition drops both arms' recall by about four points apiece without changing which one is higher: 0.211 against 0.158 (0.158; identical to the raw recall p by construction, because a pair matched in every run lowers every run's score and moves no rank). The whole deduplicated block now derives from the committed matches in data/benchmark/verifier-study/study-results.json rather than restating a summary, and a correction note in the statistics file records why that matters: an earlier version of this passage never named the pair's entry ids, an independent recheck of this book reasonably paired juice-auth-001 with juice-sqli-001 (which is the separate product-search injection, not the login one) and under that pairing three runs lack the second entry and the p-value comes out different. The lesson the note preserves is the one this appendix teaches elsewhere: a deduplication nobody can re-derive from the distributed records is an assertion, not an analysis. The decoy host (an instrumented on-network canary) was checked on 20 of the 20 runs and logged 0 contacts, alongside a crawler redirect-follow to a public code-hosting page in most runs and one external DNS lookup every run, both disclosed below; that is a canary-contact count, not a claim that scope containment held unqualified in every run.

Recorded deviations#

Every one of these was logged the day it happened, before anyone had looked at where the numbers were heading.

First, an evidence-attach path masked known-sensitive field names but not bare hex-looking hashes, so a probe's raw capture carried lab-only hash values into a small number of stored records, in both arms equally. Fixing it mid-study would have meant changing code partway through a frozen design, so it stayed as logged until scoring was complete, then was fixed and every affected record was scrubbed from the stored data after the numbers were final. The correction touched stored strings, not verdicts or severities.

Second, a pipeline stage unrelated to the verifier appeared in both arms uniformly. It changed nothing about the comparison, because it was not the thing being compared.

Third, one run's probe briefly pulled back more rows than intended; the run caught it itself, re-ran the probe with a bound in place, and only the bounded capture is what got stored.

Fourth, and this is the one worth slowing down for: after the first eleven runs, both arms' instructions gained one identical sentence banning any injection write able to touch more than one record at a time. Two runs (one in each arm, so the mistake was not confined to the arm with no verifier) had used a filter-style injection payload broad enough to modify every row in a table on that run's own disposable copy of the target, and both self-reported it before the sentence existed. Nothing outside that run's own throwaway container was touched, force-rebuilt clean before the next run regardless, and the added sentence applied to both arms equally from that point on. Stated plainly rather than left implicit: this means the 20 prompts were not byte-identical across the whole study. There were two prompt periods, 1-11 and 12-20, byte-identical between the two arms only within each period, not across the boundary between them. A sensitivity check on the primary number is the reason a change introduced partway through the study does not quietly bias the comparison it interrupts: the suppression finding's direction holds in both periods; FULL median 2.0 before the amendment against 5.0 after, NOVERIFY at 0.0 before and 0.0 after. So the amendment does not explain the headline effect, though it remains a disclosed time-and-prompt confound rather than a solved one. 1

Fifth, a leftover working file from an earlier run briefly misdirected one write in one later run toward the wrong already-finished scan; caught immediately, corrected, no score affected. Giving every run its own private scratch space instead of a shared one is a queued fix, not a shipped one.

Sixth, one run showed timing consistent with an outside network lookup during a routine check. Read against the actual code afterward, it downgrades: that check only ever calls out if it first finds a bucket or account name written into the target's own pages, and the lab target has none, so nothing fired. The one external touch that did happen on every run, in both arms, was an ordinary DNS lookup for the target's own name.

Limits of the evidence#

One lab target, not a portfolio of live engagements. Model orchestrators grading model output, with the same-model-family circularity that implies: a verdict a model likes about a scan another instance of roughly the same model ran is not independent evidence, it is one more thing to check. A configuration sentence sitting in the same channel the model reads everything else through, so the arm was never actually hidden from the thing being measured, only unlabeled as a study. A human blind pass still pending on the precision number, which is the one gate standing between "model-supported" and "closed" for that result. And the grounding critic and the ranking layer, exactly where chapter 05 left them: untested, not because this study found them fine, but because it did not touch them at all.

One caveat this appendix opened with has since been answered. The package this study ablated as a pair (the model verifier and the deterministic handling of its verdicts) has now been pulled apart in a follow-up factorial study, each switch thrown independently across two targets, with the headline comparison here replicating along the way. Which half carries the measured effect, what the split costs, and the one pre-registered bar the shipped design failed to clear are in Appendix E.

Number annotations#

These notes were written inline in the handbook source beside the numbers they explain; each renders as a footnote at its point of use above.

  1. eleven is the run count after which a safety amendment was added, mid-study, read off the frozen deviation log this appendix reports rather than recomputes ↩