Skip to content

Build your own#

This chapter is the historical build path through core/ and walkthrough/, kept as the case study's own manual. To build the current system, follow the build sequence instead; for the recommended offline starting point, run the harness lab. Its dispatcher, strict gate and deterministic verdict acceptance are separate from the model-calling verifier discussed here. Historical absence claims below refer to this chapter's case-study path, not the added lab.

The harness I want to talk you out of building takes a weekend, and it works. You write a prompt describing the phases, you give the model a tool that writes findings, you point it at a target, and it comes back with a report. Critical severity on a handful of items, a profile stating the target has no GraphQL endpoint and accepts no XML, a coverage figure, an attack-chain narrative. Every heading a client would expect.

Then somebody asks where the severities came from, and the answer is that the model chose them. Where the profile's booleans came from, and the answer is that they are dataclass defaults nobody overwrote, because the probe that would have set them was never wired up. Whether the phases ran in the order the prompt lists, and the answer is that the prompt is a suggestion and there is no record either way. Nothing failed. Nothing raised. The run is not wrong somewhere you can point at, which would be recoverable; it is unfalsifiable, and it took a weekend precisely because every part that would have made it checkable is a part that was skipped.

This chapter is the ordered version. It instructs rather than argues, which the chapters before it do instead; where a step rests on an argument made earlier, it names the chapter. Follow it in order. The order is the content: a decision taken at the first step is cheap, and the same decision taken near the last is a rewrite.

What every step names#

Each step below names the invariant it protects, stated as a property that either holds or does not; the code that enforces it, by file and by function name; a test you can run, by file and test name; and, where one exists, a committed artifact. Most of the historical enforcing code sits under core/. Where a step's does not, the step names where it does instead, as the stage machine and the evidence stage's attach-flag check both do, in walkthrough/run.py rather than core/. The model-calling verifier does not ship; its step specifies the contract and distinguishes the newer lab's synthetic acceptance guard. The orchestrator-contract step points to Appendix A, an authored contract rather than an enforcement module. The measurement step names score files and a statistics table rather than treating evaluation as a property a dispatcher can enforce.

That last anchor is what makes this chapter's promise checkable rather than aspirational. The artifacts under walkthrough/artifacts/ are the output of a driver that runs the whole sequence over committed fixtures, with no network call and no model call. Regenerate them yourself. The bare command overwrites walkthrough/artifacts/ in place, so to compare rather than replace, send the run elsewhere and diff:

python3 walkthrough/run.py --out /tmp/mine
diff -r walkthrough/artifacts /tmp/mine

A gate, tests/test_walkthrough_is_in_sync.py, holds the committed copy to that same comparison and fails naming the file that moved. So you never have to trust my description of what the machinery does. That diff is the check, and it is the one I ran to write this chapter.

What the byte gate does not do#

It freezes bytes, not correctness. A wrong figure committed next to the code that computed it is byte-stable, and the gate passes it. I checked rather than assumed. In a scratch copy of the tree I changed a counter the last stage feeds, regenerated the artifacts from the changed driver, and ran both checks: the byte gate stayed green, and test_every_gate_input_is_recounted_from_the_fixtures_it_is_counted_from in tests/test_walkthrough_stages_8_9.py failed on the arithmetic, because it recounts the figure off the fixtures instead of reading it back. Byte identity is a drift check. The numbers are pinned by separate tests that recompute them, and when you build this you need both kinds.

Step 0: decide the waist before you write a prompt 1#

The invariant: every side effect a run produces goes through a single write path, and that path grades what it is handed. Get this wrong and no later step recovers, because every control you add afterwards sits somewhere a caller can go around. The argument is chapter 02's; what follows is how to hold yourself to it.

The contract is store_protocol.py:FindingStore, a typing.Protocol under core/ that declares the members the findings consumers need and implements none of them. Its add_finding docstring carries the load-bearing sentence: a conforming store normalises the finding's evidence and governs its severity before it persists the row, so a store that skips that step silently ships the ungoverned severity it was handed. store_protocol.py:update_finding_governed is the ugly member and is reproduced ugly on purpose. It is not a coroutine while its neighbours are, because its callers invoke it as a bare statement, so an async def substitute would hand a coroutine back into a discarded expression and lose the write with no exception and no failed assertion. That asymmetry is pinned by test_the_governed_write_is_the_one_member_that_is_not_a_coroutine in tests/test_store_protocol.py, which is the shape of test to copy: assert the ugliness, so nobody tidies it into a silent failure.

This repository ships no production or persistent store under core/. Persistence is out of scope, and a half-real store sitting beside the modules that call it would be the most plausible-looking wrong thing in the tree. It does ship WalkthroughStore in walkthrough/store.py: an in-memory governing reference implementation kept deliberately outside core/ so that a reader who finds it already knows what it is. Its add_finding normalises the evidence, governs the severity, keeps the record of what the governor did, and only then appends the row.

Which makes this step's argument demonstrable rather than rhetorical, so I demonstrated it. I wrote a second store with the same member names and nothing in its write path but the append, and handed both the same finding: a wildcard-CORS finding that enters at critical carrying strong evidence, the fixture tests/test_walkthrough_store.py builds for exactly this. The governing store stored it at medium with cors-wildcard among its fired rules. The persist-only store stored it at critical, wrote no governance record, and raised nothing at all. That is the failure mode in full: not an error, a report.

Run test_the_write_path_governs_before_persisting in tests/test_walkthrough_store.py. It asserts the stored severity is medium, so a stored critical proves the write path did not grade. Then run its neighbour, test_an_empty_ruleset_is_refused_at_construction. The rule loader returns an empty list for a missing file as a documented degraded mode, and governance with no rules looks exactly like governance that found nothing worth capping, so the reference store refuses to be constructed at all rather than run quietly ungoverned. Copy that refusal before you copy anything else.

There is no artifact for this step, because the waist is not a stage. Only write, consolidate and gate reach for the store beyond recording that they ran (I walked the driver's ast to check that rather than assuming it) so the artifacts written through the waist are the ones a stage persists: findings, governance, consolidation, gate. The earlier stages hand their artifacts straight back to the driver, which is why they touch the store for nothing but the ledger. Worth knowing before you assume a waist governs a whole run: yours governs the part of the run that persists, and no more.

Step 1: write the stage machine as a document 2#

The invariant: the sequence of stages is fixed before the run starts, and the model may not reorder it. A model asked to plan its own phases will skip the expensive ones under pressure and will not tell you that it did. Chapter 01 argues the point; this is the mechanics.

Write the stages as prose first, then implement the document rather than the other way round. walkthrough/run.py is a document implemented: run_all calls its stages in a straight line, in this order: fingerprint, scope, recommend, schedule, evidence, critic, write, consolidate, gate. Each stage function appends its own name to the store's stages_run list as the last thing it does before returning its artifact, and the driver hands that same list back under _stages. So "did this stage run" is a recorded fact rather than an inference from a non-empty output, which is the whole reason to write it down: an inference from output cannot distinguish a stage that ran and found nothing from a stage that was skipped.

The tests are test_the_four_stages_emit_their_artifacts_and_record_that_they_ran in tests/test_walkthrough_stages_1_4.py, covering fingerprint, scope, recommend and schedule; tests/test_walkthrough_stages_5_7.py, covering evidence, critic and write; and tests/test_walkthrough_stages_8_9.py, covering consolidate and gate, and test_the_stage_ledger_is_the_store_s_own_list beside the first. That last one is the interesting one: it asserts _stages is the same list object as the store's own, so a driver that keeps a parallel record cannot have the two disagree about what happened.

Now the part that used to have to be taken from me, and no longer does. Those tests check that each stage is present in the ledger (the assertion is a subset test over sets, or a loop of out.stage_ran(stage), which is bare membership) and until this pass, no committed test asserted the sequence: I went looking for an ordered assertion against that ledger and found none anywhere in the tree. The order is enforced by straight-line calls in the driver, and it is visible in the ledger, which comes back from run_all under _stages and is deliberately never written to disk: the writer skips any artifact name whose first character is _, and test_main_writes_one_json_file_per_artifact_and_no_stage_ledger holds it to that, so do not go hunting for it in walkthrough/artifacts/. Transpose a pair of those calls, though, and the ledger comes back transposed with the suite still green: I did that to two pairs on a scratch copy to be sure of it, and neither a subset test nor a membership loop noticed either swap. So the ledger was evidence of what ran, and not yet evidence of what ran when, and the gap was found by running the tree rather than by any gate the tree already had. tests/test_walkthrough_stage_order.py is where that changed: test_the_stage_ledger_equals_the_declared_order_positionally asserts _stages against the documented sequence positionally, == against a list rather than <= against a set, and a transposed pair reddens it and nothing else.

Step 2: fingerprint to a profile that admits what it did not measure 3#

The invariant: a profile field is either measured or marked unmeasured, and a default never reaches a reader as a measurement. This is the step where a plausible harness starts lying, and it lies quietly, in booleans. Chapter 05 makes the general case: a thing you could not reach and a thing that is not there are different findings, and a deliverable that renders them alike is lying by omission. This step is that argument applied to a single profile field.

fingerprint.py:TargetProfile is the profile: ordinary dataclass fields with ordinary defaults. Some of those fields are only ever set by an active probe, and in this repository the active probes are withheld. fingerprint.py:probe_graphql and its siblings for WebSocket, XML acceptance and error verbosity each raise NotImplementedError("live probe withheld from this public repository"). I called them to confirm that they do.

So what does the profile say about GraphQL after a run in which nothing probed for it? It says False. That is the defect: a default rendered as a finding of absence, indistinguishable in the output from a probe that looked and found nothing. The fingerprint artifact, walkthrough/artifacts/01-fingerprint.json, refuses to publish it that way. It carries an unreachable_without_live_probes list naming has_graphql, has_websocket, accepts_xml and error_verbosity, and beside it a withheld_probe_fields block recording, per field, the probe that would have set it, the value it is currently carrying, and its passive writers.

That last key is the point of the step, and it is why the block is per field instead of a list. has_graphql and has_websocket have no passive writer (nothing under core/ assigns either one) so their False means nobody looked. accepts_xml and error_verbosity are a different state entirely: both are also assigned by fingerprint.py:_analyze_attack_surface, off a regex over a response body, so their defaults mean something looked passively and saw nothing. Not measured, and measured and negative, are different facts about a target, and a reader who flattens the two together is building the defect this handbook exists to argue against.

The artifact establishes that split by parsing the shipped module rather than by transcribing a list. The driver walks the module's ast and reports the enclosing function of every assignment to each named field, so a passive writer added, removed or renamed moves the artifact instead of leaving a stale sentence behind it. A hand-written list would have gone stale the first time core/fingerprint.py grew a writer, and nothing would have caught it. I ran that helper against the shipped module myself and got the same split the committed artifact publishes.

The tests are test_the_profile_records_which_fields_a_withheld_probe_would_have_set and test_no_withheld_probe_is_ever_called, both in tests/test_walkthrough_stages_1_4.py. Read the second before you write your own version of it: it patches the probes to record a call and asserts the recording harness fires first, because a harness that cannot catch a probe call proves nothing about a run that makes none. An assertion that a list is empty is the cheapest thing in a suite to get wrong.

Then there is test_a_passively_measured_field_is_never_also_published_as_unreachable, which guards the mirror image and is the test I would have failed to write. accepts_xml is named unreachable and has a passive writer, so a response body that trips that writer would put the same field in populated_fields and in unreachable_without_live_probes at once: each half correct on its own, the pair a contradiction, and nothing in the shape of either half to catch it. No committed fixture trips it today, so the test constructs the collision rather than waiting for one. Copy that instinct: the interesting failure of a self-describing artifact is the state where two of its own fields disagree. examples/target_profile.example.json sits alongside, a filled-in profile you can read the field set off without running anything.

Step 3: the relevance table, and the formula that ranks it 4#

The invariant: which tools run against a target is a function of the profile and a table you wrote, computed the same way every time, and the model does not get a vote. The reason to give this up is always the same: the model would choose better. Sometimes it does. You will not be able to say why it chose what it chose, and next week it will choose differently on the same input. Chapter 00's Layer 1 argues the point: scoring, not deciding.

The table is tool_recommender.py:ToolRelevance, an entry per tool declaring what that tool requires of a profile, which profile fields boost it and which penalise it. tool_recommender.py:recommend walks the table and returns a ranked list of tool, score and a reason string naming the fields that moved the score. The artifact is walkthrough/artifacts/03-recommendations.json, and the reason strings are what to read first: boosted by api_types, cors_enabled is auditable in a way a model's preference is not.

Do not read a count off that list, and do not quote one in your own writing. How many tools it returns depends on the profile, and a matching count does not mean a matching list: a profile carrying backend_language php with likely_database sqlite returns as many tools as one carrying python with mongodb, trading test_sqli for test_injection_nosql, and the tools both profiles keep are not all identical either. test_deserialize is on both lists, but not with the same score, the same reason string, or the same rank: it ranks close to the middle of the list for php with sqlite, where its reason opens MEDIUM PRIORITY (test it if time permits), and near the bottom for python with mongodb, where the reason opens LOW PRIORITY (comprehensive testing only). A count says nothing about any of that. A profile holding nothing but a target_url gets a shorter list than either, because a tool whose requires the profile does not meet is dropped before it is ever scored. Build those and run recommend over each. A figure quoted here would rot the next time the fixtures moved.

tool_recommender.py:tier labels each score, and the label is inert inside the recommender. Force it to return the skip label for everything and the ranked list comes back in the same order with the same scores and different prose; the test that pins that is test_tier_label_changes_only_the_reason_string in tests/test_chapter_claims.py. Read the tier as advice to whatever runs the list, then, not as a decision the scorer takes, which means the scheduler is where a tier has to be honoured, and if you skip the scheduler the label does nothing at all.

That scheduler is scheduler.py:SmartScheduler, and scheduler.py:adjust rescales the recommender's scores from recorded history for the current profile hash: success rates, per-profile contextual rates, a fatigue decay over consecutive failures, correlation between tools that succeeded against the same URL. The artifact is walkthrough/artifacts/04-test-plan.json, and in that run the adjusted order is identical to the recommender's, tool for tool. That is a property of this run and not of the scheduler: the driver constructs it with no stats file and never calls record, so every tool draws the same new-tool multiplier, and a uniform multiplier cannot reorder anything. Do not read that artifact as evidence the scheduler is a no-op, which is the reading its own bytes invite.

What history does once it exists is bounded, deliberately. test_fatigue_floor_binds_from_the_sixth_consecutive_failure in tests/test_chapter_claims.py drives a tool into repeated failure and shows the decay reaching FATIGUE_FLOOR and stopping there: past that point further failures move the score by nothing. A decay with no floor eventually removes a tool from every plan on the strength of one bad afternoon, and you will never watch it happen. examples/scheduler_stats.example.json is the shape of the file that history lives in, with synthetic numbers in it, so you can see what you would be persisting before you persist any of it.

Step 4: the finding schema and the evidence contract 5#

The invariant: the evidence on a finding is exactly what this module's capture functions produced from a real exchange, never a hand-built stand-in, and whether that evidence is attached to a finding at all is a declared fact, not a default. Chapter 02 opens on a finding with no request and no response in the record, indistinguishable downstream from one that has both. This step is easy to skip, because skipping it costs nothing on the day: a finding that carries a severity, a title and a paragraph of prose reads as finished whether or not the exchange behind it would survive a second look, and the gap between the two shows up to a reviewer, or to the finding's own author, a week later, by which point the request is gone, not on the day the finding was written. Evidence captured through the real capture functions is what makes the difference: it turns a finding into something a reader can re-examine rather than something they must take on faith. That guarantee has a cost, and it is worth stating rather than hiding: you cannot attach an exchange you never captured, which means the capture has to be wired into whatever runs a test before the tool that needs it is, not after.

Whether an exchange belongs on a finding at all is not a default either. walkthrough/run.py's _attach_flag reads expected.attach_exchange_evidence off the fixture that produced the exchange and raises RuntimeError unless what it finds there is a bare True or False; a missing key, None, and the string "false" (truthy in Python, so a check that merely tested truthiness would get exactly backwards) are refused rather than interpreted. Read the test's body before trusting its name. test_a_fixture_that_does_not_declare_the_attach_flag_is_refused, in tests/test_walkthrough_stages_5_7.py, calls _attach_flag directly against dicts built inside the test, under a filename that announces what it is, "99-synthetic-not-committed.json". No fixture by that name is committed to this repository. What the test proves is narrower than its name reads: that the function itself refuses a missing key, None, and the string "false", and accepts a bare boolean, not that a fixture file missing the declaration is caught anywhere upstream of it. It is not caught upstream. walkthrough/fixture_schema.py's validate_fixture (the function the whole committed fixture set is held to elsewhere in the suite) checks kind, provenance, and the request/response shape, and its body never names expected or attach_exchange_evidence; strip expected entirely from a copy of a committed fixture and validate_fixture still returns no problems. _attach_flag is where this declaration meets a check for the first time, once per fixture, from the evidence stage, and it refuses that same stripped copy outright. A schema nothing refuses is a suggestion; for this field, the schema is a single function, not the fixture format.

http_evidence.py:capture_request and http_evidence.py:capture_response build the request and response records from fields the caller holds. http_evidence.py:capture_from_requests_response and http_evidence.py:capture_from_aiohttp_response read response attributes and call those capture functions. The module imports only json and typing, so a plain test class can supply the expected attributes without either client library installed. The response side never publishes the body it captured. A truthy body becomes body_snippet, sliced to the module's cap (_MAX_BODY_BYTES), beside body_length, the true length of what came in before any slicing; a flag saying the body was cut would not say whether the cut part could have mattered, and the pair says both at once. That guarantee is capture_response's alone. test_the_response_discloses_a_clipped_body_and_the_request_does_not, in tests/test_http_evidence.py, pins the asymmetry directly: a long response body comes back with its snippet at the cap and its length reported in full, the same-length request body comes back clipped with no length published beside it, and a dict-shaped request body is serialised whole and not capped at all. Promise the whole body on both sides and the contract promises something a live run cannot keep at scale; a snippet and a length, disclosed, is the difference between a reader who knows what they are missing and one who does not know they are missing anything.

The artifact is walkthrough/artifacts/05-evidence.json, and every entry in its exchanges list carries a source naming the fixture file it came from, a request and a response built by capture_request/capture_response, and attached_to_finding, the boolean that decides whether this exchange's capture also lands inside a finding's raw_data. That boolean is computed once, by _attach_flag, and handed to the write stage as part of the same record rather than recomputed there, so the capture and the finding built from it cannot disagree about it. test_evidence_is_captured_through_the_real_capture_functions, in tests/test_walkthrough_stages_5_7.py, compares every committed entry against calling capture_request/capture_response directly on the same fixture, so a driver that reshaped what came back, or hand-built an exchange, fails here instead of shipping a capture the module never produced; it also asserts "body" absent from every response the artifact carries, checking the snippet-not-body guarantee a second way, from the committed bytes rather than from the function call. What the flag decides shows up by comparing 01-cors-wildcard.json against 07-evidence-ceiling.json directly: the first declares it true, and the finding built from it, in 07-findings.json, carries an evidence key under raw_data; the second declares it false, and the finding built from it carries no evidence key at all, though 05-evidence.json captures both exchanges the same way, as a request/response pair keyed by source. Attaching to a finding and being captured are different facts, and this is where the artifact keeps them different.

The same add_finding docstring already quoted for its severity half, back when this chapter named the waist, opens with the evidence half: a conforming store normalises the finding's evidence and governs its severity before it persists the row. test_the_governed_write_is_the_one_member_that_is_not_a_coroutine, in tests/test_store_protocol.py (the same test already used there to pin update_finding_governed's asymmetry) also asserts that add_finding and get_findings ARE coroutines, an assertion that earlier argument did not need and this one does. A coroutine function's body does not run at all until something awaits it: call one without awaiting it and you get a coroutine object back and nothing else. No exception, no evidence normalised, no row persisted, and only a RuntimeWarning naming a coroutine nobody awaited. The waist step walked through update_finding_governed failing silently by being made a coroutine it should not be. The member that is supposed to normalise a finding's evidence fails silently in the opposite direction: not by becoming a coroutine, but by being one already and never being awaited as one: one write lost by turning async when it must not, the other lost by staying async and never being waited on.

Step 5: the governor, its rules file, and the never-escalate invariant 6#

The invariant: a governing pass may lower a severity and is structurally incapable of raising one. Not a policy the pass is asked to follow: a property of how each of its comparisons is written, so a model that produced a finding cannot argue a lowered severity back up once the pass has run, and a reviewer reading a demotion never has to ask whether something upstream inflated it first. Chapter 03 makes that argument in full; what follows is the shipped mechanism it rests on, by name, and how to check the claim yourself rather than take the argument's word for it.

The enforcing code is severity_governor.py:govern_finding, in core/severity_governor.py. It reconciles one finding against three signals in a fixed order: an authored CVSS vector, the semantic ruleset in core/severity_rules.json, and an evidence ceiling computed from how replayable the finding's evidence is, and every branch across all three moves a severity only when the candidate's rank is strictly below the rank already held, with one exception: marking a finding false positive moves straight to the floor band and needs no such comparison to do it. The rules that ship, by id, are tokenization-key-public, read-via-post-bounce, spa-fallback-api-200, source-map-disclosure, csp-weakness and cors-wildcard, each an id, a match block, an action and a rationale in prose. The artifact is walkthrough/artifacts/08-governance.json.

The evidence ceiling runs last, after the CVSS reconciliation and after the rule loop (including whatever floor the false-positive branch reached inside that loop), so whatever severity survives to the ceiling is what the function returns, and nothing after it gets a chance to revisit the question. Where a reader's assumption about the ceiling usually goes wrong is the mapping itself, and the module's own docstring calls this out directly: EVIDENCE_CEILING caps thin evidence at medium regardless of what severity or CVSS vector the finding claims, but moderate and strong both map to critical, the top of the scale, which is no cap at all. Only thin evidence is actually held down anywhere in this file; build your own assumption that anything short of strong gets capped somewhere in this pass, and the assumption is wrong before you finish writing it down.

A governance record is written only when the severity or the false-positive flag actually moved; a pass that looks at a finding and leaves it alone earns no governance_record at all, on the finding or inside its raw_data. The reason is stated directly in govern_finding's own docstring: writing a record on every call, whether or not anything changed, is how a set of records ends up full of entries whose rules_fired is empty, and a count taken off those records then measures how many findings were looked at rather than how many were actually changed. test_a_governance_record_is_written_only_when_something_changed, in tests/test_severity_governor.py, is the narrow proof of the behaviour itself: a low-severity, strong-evidence finding that matches none of the shipped rules and sits inside its own ceiling comes back with rules_fired and governance_record both absent. Read that absence the way the function means it, which is not how an absent row usually reads elsewhere: it means the pass looked and found nothing to change, not that the pass never ran. Treat it as a missing check instead of a clean pass, and you will build a monitor that alarms on your healthiest runs.

Run test_the_governor_never_escalates_over_the_whole_space, in tests/test_severity_governor.py, for the invariant itself, and read past its own name before you trust it, because it claims more than its body covers. Every finding it builds carries the same fixed type, the same url and no CVSS vector at all, crossed against every severity, every evidence grade and both environments, so the rule loop and the CVSS reconciliation are never exercised by it. Only the evidence-ceiling branch is under test here, which is exactly what govern_finding's own docstring says this particular sweep reaches and nothing else. The "whole space" in the name is that cross-product of three axes, not the whole space a finding can vary over; test_no_combination_of_the_three_signals_can_raise_a_severity, beside it in the same file, is the wider sweep that actually exercises rule-matching and CVSS provenance as well, crossing a matching shape built for each shipped rule (source-map-disclosure included) against the same severities, grades and environments, plus every vector position and provenance label the reconciliation reads.

Neither test ever reads governance_record or the finding's own rules_fired anywhere in its body, so the pairing between those keys has no committed test pointed at it directly. Different conditions gate them: the record itself sits behind an if changed guard, and the finding's own top-level rules_fired sits behind a narrower if fired, nested one level inside it. Nothing in the source ties those two guards to each other on purpose. Unlike test_the_governor_never_escalates_over_the_whole_space, I built a finding shaped to satisfy each shipped rule's own match block (a different type, title, url and evidence string for each one) and pushed it through every severity, evidence grade and environment those tests already use, and did not find a run where either key showed up without the other. The reason sits a level down, in the branches rather than in the two guards themselves: the CVSS reconciliation, every rule action and the evidence ceiling each append the identifier responsible for a change in the same statement that performs the change, so none of them can move a severity, or flip the false-positive flag, without also naming itself in fired. That is a fact about how those branches happen to be written today, not a rule this module states anywhere or a property either committed sweep checks directly. I checked what that costs on a scratch copy: one more branch, gated on nothing this chapter needs you to reproduce, that moves current down without appending to fired. The pairing came apart exactly as expected: a governance_record with an empty rules_fired inside it, and no top-level rules_fired on the finding at all, and the whole committed suite still passed. Something did eventually fail on that scratch copy, but only because the comment I wrote for the branch itself used the phrase this chapter's own house rules ban, and a docstring-quantity test elsewhere in the suite caught my prose before anything caught my logic. Reworded the comment and the suite went green again, defect and all.

The records inside walkthrough/artifacts/08-governance.json are not shaped the way govern_finding itself builds one, and the difference matters if you plan to cite either as what the other produces. Every record in the artifact carries a finding_id and a source naming the fixture it came from; govern_finding's own record carries neither key anywhere. I called the function directly on a fresh finding and read back what it built, and neither name is in the dictionary it returned. walkthrough/run.py's write stage, _stage_write, adds both, for every finding it stores, whether or not govern_finding produced a record for that finding at all. Where a record exists, the driver copies it through and updates it with finding_id and source; where none exists, the driver writes an entry of its own instead: the severity that entered, a null final_severity, the evidence grade, and a note reading "no rule matched and no ceiling applied", a sentence govern_finding never writes anywhere. The entry sourced from 06-no-rule-matches.json is that second shape; the entry sourced from 01-cors-wildcard.json is the first, carrying cors-wildcard in its rules_fired on top of the driver's own two additions. So the artifact is a per-finding accounting the driver builds over the governor's narrower contract: govern_finding reports what it changed and stays silent about what it left alone, and the walkthrough writes down both, because an artifact meant to show what a run checked cannot only list what the run acted on. source-map-disclosure is the one shipped rule with no fixture of its own in walkthrough/fixtures/, so it never appears in this artifact; test_no_combination_of_the_three_signals_can_raise_a_severity is where it is actually exercised, though only for its matching and for the never-escalate invariant: appendix C measures what that leaves unheld, which is the capped value itself. Calling govern_finding yourself against a title containing "source map exposed" is how you would see it fire outside a test.

Two more tests, both in tests/test_walkthrough_stages_5_7.py, show the governor acting inside that real run rather than in isolation. test_at_least_one_finding_is_downgraded_by_the_write_path runs the whole walkthrough and asserts that at least one stored finding's final_severity differs from its original_severity, which is what proves the write path actually governed something on this run rather than merely calling code that could have; it then checks, by substring, that every record with a null final_severity carries some form of the words "no rule matched", and separately checks the one such record this artifact actually produces against 06-no-rule-matches.json's own expected.note by exact equality rather than by substring, so a near-miss in the driver's wording would pass the first assertion and still fail the second. test_the_evidence_ceiling_fired_on_the_fixture_that_cannot_prevent_a_strong_grade is named for the trap it guards against: its fixture is built so that attaching its capture the way a driver ordinarily attaches one would grade it strong and let the ceiling through untouched, so the test does not stop at checking that the run is green or that a finding exists for it. It reads the stored evidence grade back off the governed finding and asserts it is thin, asserts evidence-ceiling is named in rules_fired, and asserts the stored severity actually landed at medium, because a fixture built to demonstrate a ceiling firing is exactly the place a quiet regression would otherwise hide.

Step 6: the verifier's asymmetric raise, specified but not shipped 7#

The invariant: a verifier may lower a severity on evidence it produced itself, and may raise one only against a verbatim quote it did not write. This is the historical contract's minimum condition, not a sufficient proof policy. Lowering cannot inflate severity, but it can conceal a real issue and needs justification and counterexample tests. Raising needs finding-bound evidence and a reviewed domain predicate, not just the model's own say-so or a matching quotation. The model-calling verifier specified here remains withheld. The separate harness/ lab now ships a deterministic acceptance guard over synthetic evidence, which chapter 07 explains and explicitly does not call a production verifier.

Keep the search scoped to the component being discussed. core/critic.py reads a model's JSON verdict about evidence containment; it does not re-rate severity. core/severity_governor.py governs downward. Neither is the historical model-calling verifier. A repository-wide search now also reaches Harness.verify_raise in harness/runtime.py and its tests, so claiming that no code here constructs or checks a verdict would be false.

The historical downward half ships as severity_governor.py:govern_finding. test_the_governor_never_escalates_over_the_whole_space and test_no_combination_of_the_three_signals_can_raise_a_severity, both in tests/test_severity_governor.py, exercise the scopes described in the previous section. For the lab's separate raising path, run test_bad_quotes_do_not_raise_or_mutate, test_real_quote_without_independent_predicate_is_insufficient and test_matching_kind_tool_cannot_raise_without_capture_predicate in tests/test_harness.py. They test acceptance conditions, not a model's ability to discover or prove an exploit.

When building a real verifier, hold the finding and its captured exchange fixed and vary the quote: missing, paraphrased, or copied from another finding must not authorize an increase. A valid quote must still pass the independent proof policy. Then keep a valid quote and matching finding kind and tool while breaking each proof predicate separately; every broken predicate must refuse the raise without mutating the finding. The lab supplies synthetic examples of those tests. A domain owner must supply captured positive and negative cases for the real proof semantics before connecting a model or a live adapter.

The historical walkthrough/ has no verifier stage or verdict artifact. The lab's code, tests and harness/report.json are separate anchors for its bounded acceptance guard. That distinction lets a reader run what is published without mistaking a synthetic marker check for the missing model-calling verifier.

Step 7: the scope guard, and the defect it publishes about itself 8#

The invariant: what is in scope is computed, not judged, and the computation is auditable.

The code is scope_guard.py:ScopeGuard, in core/scope_guard.py, and its own module docstring opens by naming what the module gets wrong rather than by describing its interface. Admission is computed at the registrable domain (the last two dot-separated labels of the hostname, never the host a scan was actually pointed at) and the docstring's own first example is the structural case, not the dramatic one: a target of app.corp.example reduces to a base of corp.example, so both other.corp.example, a sibling, and corp.example itself, the parent, answer in-scope beside it, for any hostname carrying three or more labels. Only after that does the docstring reach for the sharper case: shop.co.uk reduces to co.uk, a public suffix rather than any one company's registration, so an unrelated company's site under it answers in-scope too. Chapter 04 already argued why that matters and what it costs; read the docstring itself before you copy the file, in the terms it uses for itself.

scope_guard.py:_registrable is the whole rule behind that base: the last two labels, or the whole host below two, with no public-suffix list consulted anywhere, so what it derives is a function of how many dots a hostname happens to carry and nothing else. scope_guard.py:is_in_scope takes a URL and returns a bare fact, True or False, nothing partial and nothing logged as a suggestion, computed against that base and two pattern lists the guard was constructed with, an out-of-scope list and an in-scope list, each matched by scope_guard.py:_matches as an exact host, a subdomain of one, or a shell-style glob.

Run test_the_registrable_default_admits_subdomains_of_the_target, test_the_registrable_defect_reaches_a_two_label_public_suffix_as_well and test_the_registrable_defect_is_present_and_named, all in tests/test_scope_guard.py. Read the third one's body before you write about it, because its name promises something its assertions do not deliver. It reads as though it asserts the module still says what is wrong with it; it does not. What it asserts is behavioural (a sibling host and a parent host of app.corp.example both come back True) and its own docstring pins that pair as current behaviour "so that FIXING it turns this test red deliberately." Nothing in this test, or anywhere else in the tree, asserts that the caveat text in the module docstring survives an edit. So it is a pinned-defect test in the narrower sense: it goes red when somebody fixes the admission rule, not when somebody quietly deletes the paragraph disclosing it. That is still a genuinely useful instrument, a deliberate defect with a tripwire on its own removal so the fix cannot land silently, and it is close to the opposite of what its name suggests on a first read. Copy the instinct this chapter has already asked for more than once: write down what a test's body does, not what its name promises.

One property of scope_guard.py:is_in_scope is easy to miss reading the docstring once: out-of-scope is consulted before in-scope. test_out_of_scope_beats_in_scope_for_the_same_host, in the same file, constructs a guard with one host on both lists and gets False back. Put the two checks the other way round and the same pair of declarations admits the host the operator wrote down to exclude, because an in-scope match would return before the out-of-scope list ever got read. Checked in the order the code actually uses, an explicit exclusion outranks a broader inclusion every time; checked the other way, writing a host down as excluded would stop meaning anything the moment a wider pattern also happened to cover it.

The artifact, walkthrough/artifacts/02-scope.json, carries the defect rather than hiding it. Every committed fixture URL reduces to the same base as the target, example.com, so all of them answer in-scope, and beside them sits a registrable_parent entry for https://example.com/ itself, with a top-level known_parent_domain_admission key set to true: the artifact naming its own defect rather than leaving a reader to notice it unaided. A deliberate_out_of_scope entry for a host on no fixture's list closes the artifact out at false, the ordering property above made concrete on a real run rather than only in a unit test.

Step 8: the gate check, and the fail-open the tree names as one 9#

The invariant: whether a run may proceed is a pure function of recorded inputs, replayable from them, and it names every input it consulted.

The code is gate_check.py:decide_gate_status, in core/gate_check.py. The artifact is walkthrough/artifacts/10-gate.json.

Lead with the default rather than with the range of outcomes it sits beside, because the tree names it as such and a manual that buries it is dishonest. Run test_an_empty_input_proceeds_and_that_is_the_fail_open, in tests/test_gate_check.py: an input dict with nothing in it comes back proceed, the least restrictive answer the function ever gives. The module's own docstring gives the reasoning rather than leaving it to be inferred. Every one of the function's seven optional keys defaults when the caller never set it, so an empty dict is not treated as an error; it is read as nothing having been collected yet, and it resolves to the outcome that assumes the least about what went wrong. That is a designed default with a real consequence: a gate handed nothing cannot tell a clean run apart from a run whose inputs never arrived, and on nothing at all it says proceed rather than refuse. It was built that way round because the alternative fails in the direction that matters more to a gate sitting in front of a scan: blocking on missing evidence would stop a legitimate target over a plumbing failure somewhere upstream of this function, and this tree accepts the opposite risk instead. Closing the gap would mean giving the function something to read besides absence: a per-key flag saying a signal was actually collected this run, say, or an outcome reserved for an input that never reached this call at all, and nothing here does that; today, the empty dict and a dict that collected seven honestly-measured zeroes are the same input.

The four outcomes are gated, gated_soft, limited and proceed, each naming how much of a run may still go ahead, from baseline_passive alone at the most restrictive to full at the most permissive, and test_the_four_outcomes drives all four from four built input dicts in one parametrised test. They are checked most-specific-first, and the order is load-bearing rather than a style choice. test_branch_order_is_load_bearing builds an input that satisfies both gated's condition and gated_soft's weaker one at once, and asserts the function still returns the more specific of the two: every input that trips a near-total, WAF-confirmed block also trips the plainer high-error condition sitting under it, so reading the weaker branch first would report a fully-blocked target as merely soft-gated. gated_soft and limited overlap the same way and more narrowly, and test_gated_soft_outranks_limited_on_their_overlap is built for the overlap rather than the general case: a high error rate with no parameters or forms found, pages still crawled, and, the clause that actually makes the branches coincide, since gated_soft's own condition never mentions it, no scripts found either. gated_soft wins there too. test_the_decision_is_pure_and_replayable calls the function twice against the same dict and asserts the two results equal, which is the entire content of replayable for a function with no state to disagree with itself over. test_evidence_echoes_every_input_consulted checks that all seven of the keys the function reads come back inside its own returned evidence block, so a stored decision can be re-derived from the record alone rather than trusted on the say-so of the day it was written.

The docstring goes further than the function it documents, and is honest about where the function's inputs actually come from in the system this chapter re-expresses. Three of the seven, the WAF flag and the two response-shape counts, are read from an earlier stage's artifact only if that artifact happens to exist yet, and two more of them, the form and script counts, are hardcoded to zero by the caller regardless of what a crawl actually saw, because nothing upstream reports them; only the parameter and page counts stay live no matter what. Set that beside a fact this project's own corpus can check: of 86 scans that reached this gate, 0 came back gated, the top branch. The docstring draws the careful conclusion rather than the satisfying one. It does not read that count as dead code, because the two hardcoded keys make part of gated's own condition true by construction rather than by observation. It does not read the caller as being at fault either, because a target that genuinely was never blocked that completely would produce the same count. Chapter 05 walks that same figure and the same pair of readings in full; here it is enough to know the count is real and that the caution around it is the docstring's own, not something added after the fact.

The artifact, walkthrough/artifacts/10-gate.json, is proceed on this run's own committed fixtures: crawlable, one form, one script, no errors, the richer end of the range rather than either fail-open, and its evidence block matches its inputs block key for key, which is what test_evidence_echoes_every_input_consulted checks in isolation and what this artifact then shows on a real run rather than only inside a unit test.

Step 9: the orchestrator contract, as a forward reference 10#

The invariant: the orchestrator's contract is a document, fixed before a run starts, that the model does not get to renegotiate once the run is under way.

The historical contract is Appendix A, genericized for a public reader. It is a document, not executable enforcement under core/ or walkthrough/. The newer lab freezes its capability registry and action budget in Harness; its tests exercise that boundary. Those tests do not prove that an arbitrary live orchestrator obeys Appendix A or cannot reach a tool outside the provided API.

That contract is a different document from the nine-stage walkthrough driver the rest of this chapter has been walking through. This chapter's own stages replay committed fixtures through this repository's reference modules with no model anywhere in the loop; the contract this step is about is what a live orchestrator, model included, reads before it acts, and it is not new to this book even though it has had no step of its own before now. Chapter 00 already named it in passing: "Mine is a markdown contract the orchestrator reads at the start of every run." Chapter 01 named the same kind of document again for its own stage order and was exact about the honest limit of the arrangement: it is "a contract the orchestrator follows rather than a state machine that refuses out-of-stage tool calls," which means the sequence is fixed on the page and what stops the model reordering it is the orchestrator advancing the stage on the model's behalf, not a wall the model hits if it tries. Chapters 04 and 05 each report a further piece of the same kind of document doing real work on their own subject: a skip ledger built by an agent following one, an allowed-stages table copied into one that happens to still agree with what gate_check.py:decide_gate_status computes today. None of those chapters had reason to say what belongs inside the document itself. This one does.

At minimum, on the evidence the rest of this book already gives you: the stage order, stated once rather than left to be inferred from whatever usually runs first. Which tools are reachable from which stage, because a fixed order that still lets the model reach a testing tool during observation is a fixed order in name only. The gate's own table of what each outcome still permits, kept in the same document as the function that computes it rather than copied once and left to drift, since chapter 05 already found a system where the two copies happened to agree today with nothing keeping them that way. And an instruction to reach a finished state on every run, including the ones that went badly, written as a rule the orchestrator follows rather than left to whether a run happens to reach its own last line. Appendix A is where a contract meeting all four is written out in full. This step is only the case for why one has to exist, and where a reader finds it.

Step 10: measure it, and publish what you could not measure 11#

The invariant: a claim about how well this works is a measurement against a ground truth somebody else can inspect, or it is nothing.

There is no core/ file for this step, because there is nothing here for enforcing code to be a claim about: measuring a system is not a control the system runs under, it is a practice you hold yourself to from outside it. The artifacts are score files under data/benchmark/ (harness_juiceshop_run1.score.json, run2 and run3, plus zap_juiceshop_baseline_result.score.json) and the aggregated figures in data/stats.json, which this step only reads and does not touch.

Read the score files themselves before the aggregate, because they show exactly what is and is not published. Three carry the label of an attempt by the system; the fourth is labelled as an OWASP ZAP passive scan. All four contain aggregate counts only. They do not contain raw findings, ground-truth entries, matcher identity, target identity or run identity, so equal ground_truth_count values cannot establish that the inputs or scoring procedure were the same. Author-recorded OWASP ZAP passive-scan aggregate. This repository does not publish the raw findings, ground-truth entries, matcher, target identifiers or run identifiers needed to establish an identical evaluation procedure; do not treat this row and the scanner rows as a controlled head-to-head. records this evidence limit instead of presenting the pair as a controlled head-to-head. The practice you should copy is stricter: publish inspectable ground truth, raw tool output, target and run identifiers, the matcher and its version, repeated runs, and a comparison tool evaluated by the same declared procedure.

Only one of the three attempts against this repository's own target is included in the published mean; the other two are excluded, and both are published in full, real metrics and real reasons together, rather than dropped from the file. Chapter 05 makes the case for why that split is defensible for one exclusion and only partly defensible for the other, and it is the argument to read before you adopt this practice's exclusion habit for your own runs; this step's job is narrower, which is to point at where the score files and the published split actually live and to have a reader notice the split exists at all before copying a headline off the top of data/stats.json.

And the honest part, which is the point of the step more than any figure in it. Open data/stats.json and read benchmark.juice_shop._unmeasured_reason next to the mean it sits beside: a spread over a sample of 1 is not a thing that exists, and the file says so in a key of its own rather than leaving a null stdev to be misread as a measured zero. A measurement section that publishes what it could not measure in the same file as what it could is chapter 05's argument about honest reporting, made concrete in a statistics file instead of a finding, and a reader who copies only the metrics off this file has copied the wrong part of it. Every figure in this step resolves to data/stats.json; quote it, and do not recompute a mean or a spread yourself from the score files, which is exactly the arithmetic this file has already done and published the reasoning behind.

What stays stable when the target changes#

The sequence is reusable; a web implementation is not. Do not read this chapter as a claim that a single table changes when you point the system somewhere else. The target has to be made explicit in the observation adapter and profile, the catalogue of tools and rules, the authorization and scope policy, and the evidence extractors that can substantiate a result. Those are inputs an operator can write down, diff, review, and replace. The invariants above tell you what each input must not be allowed to decide on its own.

That is the portability claim this manual can support. The ordered development path and the constraints travel; the target-bound inputs are visible instead of being hidden in a prompt or treated as universal by accident.

What the verifier study says about these steps#

None of the steps above rests only on argument now. Appendix D ran a study against two of them; the failure museum in appendix C pins several more as things the study's own artifacts, and the runs behind it, actually caught going wrong; the rest are still exactly where they were, unmeasured. Keep the three kinds apart, because they earn different amounts of trust.

Step, or the chapter that covers it What the study says
The verifier's asymmetric raise (step six) Supported: pre-report suppression and blinded shipped precision both moved with the gate, in the direction the design predicts. Appendix D.
The scope guard (step seven) Supported: the decoy sat untouched in every measured run, including the runs given no scope declaration at all. Appendix D.
The evidence contract and the write path (steps zero, four and five) Defect-corrected: a structured proof once sat beside the value the grader actually read, serialized and ignored rather than inside it, so a verified critical shipped labeled medium. Appendix C, "Two correct rules, one wrong severity".
Evidence capture on the probe path (step four) Defect-corrected: an attach path once copied a raw, unredacted capture underneath an already-redacted summary. Appendix C, "The attachment that unredacted the finding".
Consolidation and dedup (chapter 05) Defect-corrected: a correction carrying strictly better evidence was once silently absorbed as a duplicate and never reached governance. Appendix C, "Dedup ate the correction".
A memory-system kill switch Defect-corrected: two read paths and a fallback writer once ignored the flag entirely, so a study arm built on it measured nothing. Appendix C, "The kill-switch that half-killed".
The relevance table and the scheduler (step three) Untested: this study measured one control, not the ranking layer. As of the register date in appendix F, the study chapter 05 asks for is still unrun.
The grounding critic (step six's neighbor) Untested: appendix D's two arms never touched it either.

What it costs to build this#

The steps are a sequence rather than a menu, and that is the cost before any code is written. Take the governor without the fingerprint and there is nothing for it to govern; take the stage machine without the write path and the run is deterministic in the part that was never the problem. A reader who adopts only the parts that look cheap gets the parts that do nothing, and the order is what this chapter is actually offering.

After that the maintenance falls on the fixtures rather than on the modules. Every rule wants one, and a rule that ships without one has nothing holding its value: source-map-disclosure sits in the shipped rules file with its id, its position and its pattern all pinned, and appendix C measures what that leaves unheld: raise its cap to critical and nothing in this repository notices. Step six is the same cost in its other form, and it is the one this chapter cannot pay on your behalf: it ships as a specification, so the test that proves it is a test you write.

Number annotations#

These notes were written inline in the handbook source beside the numbers they explain; each renders as a footnote at its point of use above.

  1. 0 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩

  2. 1 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩

  3. 2 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩

  4. 3 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩

  5. 4 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩

  6. 5 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩

  7. 6 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩

  8. 7 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩

  9. 8 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩

  10. 9 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩

  11. 10 is this step's ordinal in the chapter's own sequence, a label on a section rather than a measurement of anything ↩