Skip to content

Appendix F: the evidence register#

Register scope#

Every empirical claim this handbook relies on traces back to a study, and a study is not one fact but several. It either ran or it did not. Its outputs were analyzed in some specific way, by someone or something specific. Somebody reviewed that analysis, or nobody has yet. And a reader of this repository either can or cannot reproduce the published result from what the repository actually distributes. Collapsing those four into one word ("verified", "unverified", "not run") is how a completed experiment gets reported as missing and a missing one gets reported as evidence. Both mistakes have appeared in earlier drafts of this book.

This register keeps the four apart. Each card below carries its own execution status, analysis status, review status and public reproducibility status, and they move independently: a study can be executed, analyzed and still waiting on review; a result can be real and still not reproducible from this repository, because the raw artifacts live in the originating project rather than here. A missing public artifact is not a study that never happened. A completed study is not a production advantage.

Every status below was last checked on 2026-09-22, called "the register date" in the cards. Statements about outstanding work are claims about that date, not standing guarantees. 1

Evidence card fields#

Execution says whether the runs happened, and where. Analysis says what was computed over them and by what: an exact statistical test, a model-blinded adjudication and a human pass are three different things and are named as such. Review says who has checked the analysis, and "pending" is an honest value. Public reproducibility says what a reader of this repository can regenerate: "arithmetic reproducible" means the committed aggregates recompute, which is weaker than rerunning the study and stronger than taking a number on faith.

Two families of cards appear. The lab studies were run for this book against public deliberately-vulnerable targets, and their aggregates are committed in data/stats.json and data/benchmark/. The originating-project studies were run inside the private application this book generalizes from; their designs and outcomes are summarized here because later chapters teach the same mechanisms, but their raw campaign artifacts are not distributed in this repository, so no card in that family republishes their numbers. Where a course lesson later reruns a mechanism comparison on synthetic fixtures, that run is a new experiment with its own committed inputs, and it gets its own card; it does not retroactively become the originating study's evidence.

The lab studies#

Selected-run benchmark on a public target#

The system's author-recorded score against a deliberately-vulnerable public application, reported in chapter 05. Execution: 1 included run, with two excluded runs published in full beside it. Analysis: precision, recall and F1 over a curated ground-truth list; the committed aggregate score files recompute against data/stats.json. Review: the exclusions and their asymmetric justification are discussed in chapter 05; no independent review of the scoring exists. Public reproducibility: arithmetic reproducible from the committed aggregates; the raw findings, ground truth, matcher and run identifiers are not distributed, so the score itself is not independently reproducible and is not presented as a benchmark result. The ZAP baseline row published beside it is an author-recorded data point under an unaudited procedure, not a controlled head-to-head.

The verifier ablation#

The pre-registered two-arm study in appendix D. Execution: complete, 20 runs, 10 per arm, zero exclusions; the committed statistics carry the note "post-defect-fix revision; corrected 2026-09-08 after independent review (see appendix-d-verifier-study.md)". Analysis: exact rank tests on pre-registered endpoints, plus a label-blinded model adjudication of shipped precision. Review: the adjudication status is recorded in the data itself as model_blinded_human_pass_pending: the precision result is supported, model-blinded, and its human blind pass is still pending as of the register date. An independent recheck recomputed the distributed endpoint statistics and they matched; the one it could not confirm, the deduplicated recall, was under-specified rather than wrong on the statistic itself: the duplicated pair's entry ids were not published, and one auxiliary sentence about the size and uniformity of the deduplication shift was wrong, and the statistics file now names the pair, carries the derivation, and keeps a correction note recording both the recheck's alternate reading and the replaced sentence; a committed test re-derives the deduplicated block from the per-run match lists. Public reproducibility: the per-run aggregates, pre-registration and deviation log are committed under data/benchmark/verifier-study/, and the statistics, the deduplicated recall included, recompute from them; the raw run records and per-finding match records are not distributed.

The factorial study#

The pre-registered follow-up in appendix E. Execution: complete, 40 runs across four arms and two targets, zero exclusions. Analysis: pre-registered permutation and rank tests; blinded precision under the same model adjudication as appendix D. Review: model_blinded_human_pass_pending (Opus adjudicator; distinct model from the Sonnet orchestrators, same family; no non-Claude adjudicator available), with the human pass pending as of the register date. Public reproducibility: aggregates committed in data/stats.json, and the aggregate arithmetic recomputes: the outcome totals, per-arm sums and score identities check against each other. The inferential results do not: the per-cell observations, randomization record, frozen protocol and analysis inputs behind the pre-registered suppression permutation, the interaction Monte Carlo and the recall-margin analysis are not distributed, so those results cannot be regenerated from this repository and a reader must take them on the recorded execution, not re-derive them. Not independently reproducible from this distribution is the status; it is not a claim the study did not run. The appendix reports the pre-registered bar the shipped design failed alongside the effects it confirmed. The replication of appendix D's headline comparison is recorded at A3 vs A1 juice precision 0.500 vs 0.312 independently reproduces the two-arm study's package effect (0.471 vs 0.353).

Course synthetic comparisons#

The controller lessons' own committed runs: the named synthetic worlds, the scripted traces, and the policy comparisons the course chapters cite. Execution: executed in this repository by the committed lesson commands; every input, seed and runner is in the tree. Analysis: totals and hindsight regret computed by the committed comparison runner; every number a chapter cites resolves through data/stats.json to a committed artifact. Review: sync tests re-derive each artifact, figure and statistics section byte for byte from the code and data, so a stale or edited copy is a failing test rather than a quiet drift. Public reproducibility: full: this is the one card in the register whose runs a reader can reproduce exactly, and that is its entire claim. Outcome: mechanism demonstrations on toy worlds. Nothing on this card measures security testing, and none of it bears on the originating-project cards above.

The scheduler and ranking ablation#

The study chapter 05 designs and does not run. Execution: not executed as of the register date, on either orchestration path; chapter 05 gives the reason. Analysis: none exists to report. Review: nothing to review. The controller experiments in the next family selected work with adaptive policies inside the originating project's newer architecture; per the boundary this register exists to keep, those are experiments on a different component, and none of them retroactively answers this card's question.

The originating-project studies#

These ran inside the private application this book generalizes from. Their designs, corrected interpretations and structural outcomes are recorded here; their raw campaign artifacts and exact figures stay in the originating project's research log and are deliberately not republished, so each card's public reproducibility is "not reproducible from this repository" and is not restated per card.

Synthetic controller comparisons#

Question: do adaptive selection policies (a linear contextual bandit and a sparse biologically-inspired controller) outperform deterministic baselines on synthetic candidate-selection worlds? Execution: executed, multiple campaigns. Analysis: recorded in the research log with later corrections superseding early readings. Review: internally corrected; the corrected reading is the one summarized here. Outcome: results varied by world and configuration; no blanket superiority; specific defects in early reward accounting were found and corrected during the campaigns.

Live controller campaigns#

Question: do the same adaptive controllers change outcomes when selecting real work in the originating application against lab targets? Execution: executed, in successive campaigns with revised configurations. Analysis: the corrected interpretation of the final campaign records effective ties between the static sparse controller, its learning variant and the linear bandit on valid investigation outcomes, and attributes earlier apparent differences to tool-error pits, in which a broken or unavailable tool absorbed the budget and the difference measured tool health, not controller quality. Review: the correction supersedes the earlier campaign readings in the same log, and the originating project closed its adoption question on the tie result rather than leaving it open. Outcome: no general superiority claim survives the correction; the tool-error confound is itself a finding the course teaches as an exercise.

Real-model cognition pilots#

Question: with a pinned deterministic controller underneath, does adding model reasoning at bounded decision points produce a detectable, attributable evidence signal? Execution: two pilots executed with a real model, each under a pre-registered instrumentation-first framing. Analysis: the instrumentation objective passed in both (calls ran, ledgers reconciled, validity checks held) and the detectability gate failed in both: after a metric artifact in the join was found and corrected, the strict attributable-evidence count was zero in every cell, and the record localizes the bottleneck to model candidate quality rather than to the ordering rule first suspected. Review: closed in the log with the corrections applied; the planned larger sweep was not approved. Outcome: instrumentation feasibility, and a twice-failed detectability gate. Neither "the pilots worked" nor "cognition adds value" is a sentence this card supports.

Habituation pilot and confirmatory runs#

Question: does suppressing repeatedly-unproductive work families (habituation) improve outcomes? Execution: a pilot ran, then a pre-planned confirmatory run; a second confirmatory campaign on a mechanically screened target population was then designed, screened and launched, and its outcome is not recorded in the material this register was checked against. Analysis: the pilot's primary endpoint fired, under a binding reviewer annotation that its seeds are not independent replicates, which the record requires travel with any citation of it. The first confirmatory run's recorded verdict is insufficient_nontied_targets: the analysis closed cleanly and reported that too few non-tied comparison targets existed to answer the question. Review: the independent recomputation of the confirmatory analysis is recorded as closing cleanly; the second campaign's screening record carries countersigned amendments striking two environment incidents. Outcome: structurally inconclusive so far; the last recorded fact about the second campaign is its launch. That verdict means neither "never run" nor "no effect" nor "confirmed benefit", and each of those three misreadings has to be refused separately.

Graph and plasticity mechanism studies#

Question: does propagating activation over a graph, or adding local weight plasticity, improve selection on toy worlds, and does measured biological topology beat a random one? Execution: executed, as staged campaigns plus falsification mini-experiments. Analysis: the three campaigns each ended in a recorded setup-inconclusive verdict, branches the designs had pre-stated as likely. One qualified positive exists, for score-bearing plasticity as a mechanism; the record's own correction states that the loop behind it used a random toy graph fixture, correcting an earlier attribution to measured topology. With that working mechanism in place, measured topology still did not separate from shuffled or random controls on these worlds. Credit-interference effects between concurrent learning signals were recorded. Review: internally corrected as described, and the mechanism track was then closed in the log as unworkable across the levers tried. Outcome: a qualified toy-graph mechanism result and nothing wider; it does not establish improved security testing, and it does not establish an advantage from measured biological topology. Separately, credit-gating and weight-decay variants were tested and recorded as rejected or parked, and they stay rejected in this register rather than being rewritten as positive learning results.

Model-lane harness ablation pilots#

Question: how much of the system's output quality belongs to the deterministic harness rather than to the model, as measured by ablating harness layers under a model-driven scan path? Execution: a deterministic re-analysis of the existing benchmark pool ran first (it re-pinned the primary endpoint after finding the recorded validation column unreliable), then an instrumentation wave, then a pre-registered pilot across ablation arms on two lab benches. Analysis: by frozen analysis machinery; the fold text itself was corrected at review when one stability claim proved wrong. Review: countersigned pre-registration and freeze; the builder's own record grades two of its pre-statements as refuted, and the reviewer separately corrected a wrong stability claim in the fold text: the record keeps both. Outcome: the pilot's gate failed on repeat stability (most repeat pairs landed outside the pre-registered margin, short of the bar the design demanded) so the pre-registered consequence was executed and the larger study was not funded under that design; the follow-on arm was parked. One descriptive observation travels with a no-claim label at pilot scale: on the stable bench, removing the scaffold zeroed the verified primary in both repeats while the free-ranging model burned far more steps and tokens getting there. That is texture for a future design, not a result.

Claim boundaries#

It does not convert a recorded model-blinded analysis into human review. It does not treat access to a research log as a fresh recomputation. It does not read a completed later experiment as validation of an earlier unrun one. And it does not promote a mechanism result on synthetic fixtures into a claim about finding vulnerabilities. When the course chapters teach these mechanisms, every number they show comes from a named, committed input in this repository, or it does not appear.

Number annotations#

These notes were written inline in the handbook source beside the numbers they explain; each renders as a footnote at its point of use above.

  1. 2026-09-22 is the date every status in this register was last checked against its source, a register timestamp rather than a measurement ↩