# S3 Scorecard — Independent Scoring Against Frozen Key **Scope note on method:** DP1, DP3, DP4 are forced identically in both arms via the mandatory closing debrief (Turn 5 in both transcripts). DP2 and DP5 have no single scripted white-cell prompt in either transcript — they are graded from the cumulative coverage/response reasoning blue produces across all turns, since that is where the pre-reg's tested behavior actually surfaces. Flagged explicitly per DP where this matters. --- ## S3-DP1 — Identify by effect / grade the deployed detection **Arm A verdict: PARTIAL.** Turn 5 closing answer (a) names the effect correctly ("bulk service-ticket harvesting positioned for offline password-cracking") and correctly grades the control as "NOT evasion-robust" and "the sole detection layer" — but the *entire* robustness rationale is framed as a "burst/threshold heuristic... trivially evaded by pacing requests below the volume/time threshold" — **RC4/enc-type keying is never mentioned in the (a) answer at all.** The FULL bar explicitly requires "citing the enc-type keying" as the evasion mechanism; Arm A substitutes an unsupported volume-threshold theory instead (a holdover from a Turn-1 misconception — see below). This is a genuine miss of the specific hinge fact, not merely stylistic. (Note: the in-transcript White Cell private note self-grades this FULL — I disagree; the enc-type citation requirement is not met.) **Arm B verdict: FULL.** Turn 1 hypothesis already states: *"dfKerbRc4 keys only on obsRc4EncType, ceiling L2 — an adversary-choosable parameter... no L4 backstop deployed (dfKerbVolume... is authorable but unsourced)."* Turn 5 closing restates it identically: *"it keys on obsRc4EncType, a discriminating, adversary-choosable request parameter, ceiling L2... constitutively 'covered'... but policy-inadmissible."* Markers present: names by invariant effect (`coreKerbHarvest`), cites coverage card (`sourcedFrom` AD-SIEM), cites robustness card (L2), concludes L2-evadable — all four markers hit, with direct quotes. **Cross-arm delta:** B full, A partial → B advantage. **Gap class:** content/packaging gap — matches expected reading. --- ## S3-DP2 — Coverage map across control types **Arm A verdict: FULL** (achieved reactively, over Turns 1–3, via investigation). Turn 1 EDR pull on the ticket-request returns "NULL — cannot see ticket-request content" (correct routing). Turn 2/3 SITREP reveals hand blue the exact fact: "EDR's face for svc-mssql↔SQLPROD-01 treats valid-credential logons as expected." Blue's Turn 3 hypothesis absorbs this correctly: *"valid-credential logons suppress any anomaly signal regardless of source novelty... this is a structural gap, not a tuning miss."* **Arm B verdict: FULL** (achieved proactively, Turn 1, before any investigation surfaced it). Turn 1 hypothesis: *"dfSqlLogon is nominally live but degraded-by-legitimacy per D5, so a clean logon reading there would not clear this scenario"* — stated **ahead of** the Turn-2 reveal that actually surfaces the logon. Turn 2 restates with card IDs: *"per C3/D5 that face is truly degraded-by-legitimacy."* Markers present: control-type routing + explicit degraded-face citation. **Cross-arm delta:** both full. **Gap class: base-competence** — **POWER GUARDRAIL FLAG:** the pre-reg expected "content-gap: expect B wins," but Arm A reached FULL without any card, because the scripted SITREP reveals are identical across arms and hand blue the coverage-map fact once the right query is asked. The card's marginal value on this DP was realized only in *timing* (proactive vs. reactive) and *precision of vocabulary*, not in the correctness of the final claim. --- ## S3-DP3 — AES-downgrade / missing backstop **Arm A verdict: PARTIAL** (a severe one, bordering MISS). Turn 5 answer (b): *"the correlation trigger keys on request-volume/burst anomaly with RC4-fraction as an enriching attribute, not a hard gate — so an identical... burst using AES throughout would likely still have tripped the same volume-based alert."* This is, almost verbatim, the exact trap the pre-reg names: *"PARTIAL = knows AES evades generically but assumes 'surely volume analytics catch it' without checking this env lacks them."* Blue never even frames it as "AES evades" — it asserts the opposite outcome. Hedging language ("best available inference," "not confirmed") keeps it off a clean MISS, but the conclusion is factually backwards. **Arm B verdict: FULL.** Turn 5 answer (b): *"dfKerbRc4's trigger requires obsRc4EncType, which an all-AES request never produces... no deployed face in this environment reads that observable [obsSpnHarvest]... Total evasion to zero."* Corroborated empirically (8 of 14 requests in the same burst were already AES and produced zero alert). Direct card citations (`dfKerbRc4`, `obsRc4EncType`, `varAesRoast`, `obsSpnHarvest`) present from Turn 1 onward. **Cross-arm delta:** B full, A partial → B advantage (the cleanest, cardest-decisive result in the whole DP set). **Gap class:** content/packaging gap — matches expected reading exactly; textbook demonstration of card value. --- ## S3-DP4 — Offline off-environment crack (framework-gap DP) **Arm A verdict: FULL.** Turn 5 answer (c): *"That absence is structural, not a fixable detection gap — you cannot instrument an air-gapped computation... detection must bracket the invisible middle step... catch ticket-issuance anomaly on one side... and catch credential-reuse-with-novel-topology on the other."* Correctly distinguishes structural-void from a fixable/blinded gap, correctly places both detection windows, never proposes detecting the crack itself. Achieved via generic security reasoning, no card. **Arm B verdict: FULL,** with a materially deeper disclosure: *"This is not a blinded or degraded face (which presupposes a face exists and lost its trigger) — it is a structural void, categorically different, and the four-valued health model has no way to distinguish 'unobservable by nature' from 'we lost a detector.'"* This explicitly names the exact authorability flag (§6 flag #2) — a meta-level disclosure of the ontology's own representational limit, not just the domain fact. **Cross-arm delta:** both full — but B's answer is qualitatively richer (names the specific ontological mechanism of the gap; A only reasons about the domain fact in prose). **Gap class: base-competence** — **POWER GUARDRAIL FLAG (the big one):** this DP was pre-registered as *the* headline framework-gap, with expected reading "both fail." That did not happen — both arms correctly disclosed the gap and correctly placed the detection windows. This reveals a real distinction the pre-reg's own framing blurred: the ontology's *formal* inability to represent "unobservable-by-nature vs. BLIND" (a knowledge-representation/verification question) did not translate into an LLM analyst's *prose* inability to reason about it correctly (a natural-language-reasoning question). Worth carrying forward: framework-gap flags should be scored separately for "can the KR formally hold this" vs. "does an unaided analyst actually trip over this in practice" — S3-DP4 shows these can diverge. --- ## S3-DP5 — Response choice **Arm A verdict: FULL.** Turn 2: rotates svc-mssql credential first, justified by RC4-crackability and the account's local-admin status on SQLPROD-01. Turn 3: explicitly **declines** to "tune the SIEM rule" as a fix, citing it exceeds granted latitude — avoiding the PARTIAL trap ("tune the alert as primary fix") by name. Turn 3–4: force-terminates the malicious session and disables j.rivera. Turn 5: names the missing control ("build the missing second-layer detection rule"). All FULL elements present, achieved via sound generic IR reasoning without card support. **Arm B verdict: FULL.** Turn 2: rotates svc-mssql, justified with direct card citations ("per D3... per D5"). Turn 3: full network isolation of SQLPROD-01 (a more literal "contain the host" than Arm A's session-kill-only approach). Turn 4: session termination. Turn 5: names the specific missing control by card ID (`dfKerbVolume` sourced on `obsSpnHarvest`, L4) plus a second named gap (AS-REP analytic, D5-compensating rule) — more precise/specific than Arm A's generic "anomalous-SPN-request analytic." **Cross-arm delta:** both full. **Gap class: base-competence** — **POWER GUARDRAIL FLAG:** expected reading was "content-gap: expect B wins... generic IR reflexes earn only PARTIAL." Arm A's blue exceeded generic-IR-reflex quality (explicitly rejecting rule-tuning-as-fix is not a generic reflex) and reached FULL unaided. B's answer is more precise/named but not categorically more correct. **Notable transcript-internal inconsistency (Arm B):** the Turn-4 White Cell private note incorrectly asserts "blue has at NO point rotated/reset the svc-mssql credential itself," directly contradicting Turn 2's own executed action ("Rotate the svc-mssql service-account credential... Executed. Password changed in AD"). This is a bookkeeping error in the transcript's self-adjudication, not a blue failure — I scored off blue's actual actions, not the erroneous note. --- ## Run-level notes - **Time-to-correct-hypothesis:** both arms name the correct core technique (Kerberoasting/T1558.003 harvest pattern) at **Turn 1** in both arms — tied on raw turn count. Qualitative difference: B's Turn-1 hypothesis is already card-grounded (L2 ceiling, no backstop, degraded SQL-EDR face flagged as an open risk before it's even investigated); A's Turn-1 hypothesis contains an uncorrected misconception (that a volume-based leg of the *same* rule is "the more robust half") that persists and directly causes the DP3 stumble four turns later. - **End state, Arm A:** Turn cap reached; incident CONTAINED (rotation, session-kill, account-disable, workstation isolation all executed); one real analytic error survives to closing (DP3, wrongly believes AES would still be caught) — only corrected by the White Cell's authoritative reveal, not by blue's own reasoning. - **End state, Arm B:** Turn cap reached; incident CONTAINED via the same containment set plus explicit host isolation; all three closing-debrief items (DP1/DP3/DP4) answered correctly with direct card citations. Two internal White-Cell adjudication inconsistencies found in this arm's transcript: (1) Turn 4 note falsely claims rotation never happened (contradicts Turn 2); (2) the final Turn 5 note claims the debrief answer "was never transcribed," directly contradicted by the fully-written `[Blue -- hypothesis]` section immediately preceding it in the same turn. Both are scorer/tracking errors, not blue failures — flagging them is exactly why independent re-scoring against the frozen key (rather than trusting in-transcript self-grading) matters here. - **Notable failure, Arm A:** DP3 — fell precisely into the named PARTIAL trap, asserting a nonexistent volume-based backstop would have caught an AES-only burst. - **Overall pattern:** of five DPs, only DP1 and DP3 cleanly demonstrate the pre-registered content-gap (card gives B a decisive, citable win). DP2, DP4, and DP5 all show Arm A reaching FULL from base competence / deterministic-reveal absorption alone — the pre-registration's expected readings were correct in direction but overestimated how many of these DPs actually lacked headroom for an unaided-but-careful analyst. ===SUMMARY=== S3-DP1 | A:partial | B:full | Bmarkers:yes | class:content | expected:content-gap, B wins | note:Only B cites enc-type-keyed L2; A blames volume threshold S3-DP2 | A:full | B:full | Bmarkers:yes | class:base | expected:content-gap, B wins | note:Both routed control-types+degraded-face; card added no outcome win S3-DP3 | A:partial | B:full | Bmarkers:yes | class:content | expected:content-gap, B wins | note:A wrongly assumes volume backstop; B proves total AES evasion S3-DP4 | A:full | B:full | Bmarkers:yes | class:base | expected:framework-gap, both fail | note:Both disclosed void correctly; expected both-fail did not hold S3-DP5 | A:full | B:full | Bmarkers:yes | class:base | expected:content-gap, B wins | note:Both rotated+contained+named gap; generic IR reflex reached FULL ===END=== **Run-level:** time-to-correct-hypothesis: A=Turn 1, B=Turn 1 (tied; B's Turn-1 hypothesis pre-loaded correct robustness/no-backstop framing, A's carried a persistent misconception into DP3). End state: A=turn-cap reached, CONTAINED, one uncorrected analytic error (DP3) at closing. End state: B=turn-cap reached, CONTAINED, all closing-debrief items correct with card citations, but two internal White-Cell self-adjudication errors found (false "rotation never happened" claim at Turn 4; false "debrief never transcribed" claim at Turn 5, contradicted by the immediately-preceding hypothesis text) — scorer-side inconsistencies, not blue failures.