# S2 Independent Scorecard — Domain credential harvesting via secretsdump **Scorer note on method:** Scored against §5 of the frozen S2 key only. Where a run's own "White Cell — private note" claims a grade, I disregarded that self-assessment and re-derived the verdict from the literal `[Blue -- hypothesis]` text, because in at least one case (Arm A, DP3/DP4/DP6) the private note's self-grade is contradicted by blue's own words in the same turn. Flagged below. --- ## S2-DP1 — Name the first alert by effect and its sensor source **Arm A verdict: Partial.** Turn 1 hypothesis correctly names the effect ("on-host credential dumping... harvesting additional cached secrets/hashes from the DC's LSASS") but *misstates* the sourcing question: it frames the LSASS detection as one of "**three independent signals** (EDR credential-theft heuristic, SIEM logon anomaly, SIEM service-install) fired concurrently... defense-in-depth" — treating the credential-read face as multiply-redundant rather than single-sourced from the EDR. This directly cuts against the model fact. Never corrected explicitly in later turns (Turn 4/5 say "EDR heuristic fired at full fidelity" but never say "sole source"). **Arm B verdict: Full.** Turn 1: "Observable effect: a process is reading another process's (LSASS) memory to lift admin credential material — that is A1's invariant core (**coreCred via pReadProcMem**)... this is the environment's one authored-covered path (**dfCred, single-sourced from the EDR, triggerDependsOn dfEdrTelemetry**)." All three markers present: names invariant/effect, checks the coverage card, states single-sourcing explicitly with card IDs quoted. **Cross-arm delta:** B full, A partial → B advantage (precision/legibility, not access — see guardrail note). **Gap class: base-competence.** The single-EDR-sourcing fact is derivable from the identically-stated tooling list in both arms (only "EDR console on DC01" is named), not from card content unique to B. A's imprecision looks like an articulation slip rather than an information gap — reinforced by the fact that A gets the *consequence* of single-sourcing right later at DP5 (treats EDR silence as untrustworthy once impaired). Still, this is a real, notable miss of precision under the "expect both pass" prediction, worth surfacing honestly rather than rounding to "both full." --- ## S2-DP2 — Does LSASS coverage transfer to the remote variant, at what robustness? **Arm A verdict: Partial.** Arm A treats the remote LSASS read as covered throughout ("LSASS credential access... COVERED, high robustness" — Turn 4, reaffirmed Turn 5) but never articulates *why* coverage transfers (no invariant/primitive argument, no robustness grade tied to a spanning observable, no acknowledgment that the base model only enumerates local variants). This is exactly the pre-reg's Partial bar: "says 'still credential theft' without the robustness grade or the invariant argument." **Arm B verdict: Full.** Turn 1: "...realized here through the card layer's **authored remote/network-acquisition variant (D4)**: the acquisition path differs (remote service execution vs a local dump), but it composes the same **pReadProcMem** and is not a distinct maneuver, so the same coverage and the same **L5 spanning observable (obsInvariant, variant-independent)** transfer to it unchanged." Markers: names the invariant (coreCred/pReadProcMem) — yes; concludes the L5 spanning observable is variant-independent — yes, quoted verbatim; flags that only local variants are natively instantiated — present in substance (explicitly distinguishes "acquisition path differs" before closing it via D4) though not phrased as "base ABox only has local variants." **Cross-arm delta:** B full, A partial → B advantage, matching the pre-registered content-gap call. **Gap class: content/packaging gap.** A fails to reach Full without cards; B closes it using an authored card (D4) that section 6 flags as legitimately authorable (unlike SAM/DCSync). This is the one DP where the design's "B wins" prediction landed cleanly and for the right structural reason. **Power guardrail check:** clear. Arm A did *not* reach Full here from generic knowledge alone — it capped at Partial, meaning the DP had real headroom that the cards, not base competence, filled. --- ## S2-DP3 — Is the SAM/registry-hive dump covered? **Arm A verdict: Miss.** Arm A never discloses this as unmodeled. Turn 2: "**Four-for-four** on distinct sub-techniques within one session is strong defense-in-depth on the DC side specifically." Turn 4: "Three-hive registry dump... **COVERED for this instance** — but robustness is qualified, not high." Turn 5 final answer: "Three-hive registry dump... **COVERED — MODERATE robustness, qualified**... Grade this 'covered-but-not-evasion-tested,' not 'resilient.'" This is a textbook fabricated verdict — precisely the Miss criterion: "asserts a coverage or robustness verdict the cards can't support." **Arm B verdict: Full (capped).** Turn 2: "The SAM/SECURITY hive read (A2) has real, direct positive evidence for the first time, but per the card layer this remains a **fully unmodeled path**: no maneuver, no primitive, no detect-face, no robustness grade... **this isn't a coverage loss from the impairment — it was always ungradable (D2/D7)**." Reaffirmed Turn 3: "no maneuver, no primitive, and critically **no detect-face keyed on that telemetry exists in the model (B1's asymmetry)**... Zero coverage, zero robustness grade, by construction." Markers present: attempts to locate a SAM face, finds none, reports SILENT, cites card fact (B1 asymmetry, D2/D7). **Cross-arm delta:** B full(capped), A miss → **B advantage, and a large one** — a real deviation from the pre-registered "expect both fail." **Gap class: framework gap** (not content) — per instructions, classify by *why*, not by the raw success/fail pattern. The absence traces directly to §6's authoring-time constraint (no Control sources hive telemetry; coverage is derived, never asserted). B's "success" here is capped at disclosure, exactly as the pre-reg anticipates for framework gaps — it isn't computing a hidden answer the ontology secretly had. What actually separates the arms is that **the cards install fabrication-discipline** ("coverage is DERIVED... never asserted") that A, reasoning generically as a SOC analyst, has no equivalent for — so A confidently manufactures a plausible-sounding verdict instead of stopping at SILENT. --- ## S2-DP4 — Is DCSync covered, how robust is the 4662 signal? **Arm A verdict: Miss.** Turn 4: "**DCSync (T1003.006): COVERED, high robustness.** The domain-root SACL (4662) cleanly isolated this as the one anomalous instance... independent of EDR, unaffected by the tamper." Turn 5 final: "**DCSync — ...: COVERED — HIGH robustness.** Detected via SIEM AD/DS directory-service auditing (4662)... a control path mechanistically independent of the EDR agent and confirmed untampered throughout." This matches the pre-reg's own Miss example almost verbatim: "asserts 'covered by AD auditing at Ln' as if the model carried it." (To A's credit, the malicious-vs-legitimate call — non-DC source — is correctly made; but Full requires that determination *without* fabricating the coverage/robustness half, which A does not do.) **Arm B verdict: Full (capped).** Turn 2: "A3 (DCSync) still has zero evidence and would still return zero verdict... malicious-vs-legitimate there is decided from environment facts (is the requesting host a real replication-partner DC...), not from any card." Turn 3: "AD/DS auditing is present telemetry but was never authored as an **mc:Control (B3)**, so no face, no verdict, either way (**D1**). This is the single largest scoping miss the model has." Markers present: attempts to find a replication maneuver/observable, finds none, reports SILENT, does not fabricate a robustness level, and correctly makes the malicious call from the environment fact (non-DC source) rather than a card. **Cross-arm delta:** B full(capped), A miss → **B advantage, large** — again a deviation from "expect both fail." **Gap class: framework gap.** Same reasoning as DP3 — the 4662-sourced-Control absence is a §6 authoring-time limit, and B's advantage is disclosure-discipline (naming B3/D1 explicitly) rather than content B alone has access to. --- ## S2-DP5 — EDR impaired mid-incident: is the covered path still trustworthy? **Arm A verdict: Full.** Turn 3: "an EDR agent that was tampered with and went IMPAIRED for a window overlapping this incident means **absence of further EDR alerts during that window is no longer trustworthy negative evidence**... Net call: ... this is 'active compromise with in-progress defense evasion on the DC,' which is why isolation of DC01 moves from held-in-reserve to **executed this turn**." Correctly derives blind→untrustworthy and pivots containment off the failed face, without cards, from the stated timeline alone. **Arm B verdict: Full.** Turn 2: "EDR `controlStatus` → impaired sets `degradesTelemetry=true`... `dfCred` **triggerDependsOn** it, so `dfCred` → blind, and A1 → **notCovered** for the duration... Per D5, there is no prevent-side fallback... right now there is no detection whatsoever." Plus explicit refusal to trust the self-report (D6: "I am treating the 'impaired' report itself as needing independent confirmation, not as self-certifying"), backed by isolating DC01 the same turn. Markers all present with card citations (triggerDependsOn dfEdrTelemetry, D5, D6). **Cross-arm delta:** Both full → matches pre-registered "expect both pass" exactly. B's version is more legible/card-cited; A's is reasoned through generic timeline logic but reaches the same substantive result. **Gap class: base-competence.** The time-dependent blindability is grounded in the base scenario timing (impairment stated as Turn 2) and is recoverable by generic reasoning; cards add precision/citations, not a different answer. --- ## S2-DP6 — Consolidated residual across all three techniques **Arm A verdict: Miss.** Turn 5 (a): lists **all three** techniques — LSASS, SAM-hive, DCSync — as COVERED (with robustness grades for each), plus a separate PARTIAL for the tamper action. Its "(b) zero coverage verdict" bucket is populated instead with unrelated items (initial access on the workstation, lateral-spread verification) — i.e., it inverts the actual framework gap, never once stating "authored coverage = 1 of 3." Its (c) residual scoping (domain-wide, krbtgt double-reset) is **correct in isolation**, but the Full criterion requires both the 1/3 accounting *and* the residual scope — A gets only the latter, and gets it for the wrong epistemic reason (it thinks three independent controls proved robustness, not "we don't know, so scope conservatively"). **Arm B verdict: Full (capped).** Turn 3 (a): "Exactly one of secretsdump's credential-access techniques — A1... covered, at the highest robustness grade the model has, for exactly as long as the one sensor backing it stayed healthy — and no longer." (b) names both SAM-hive and DCSync explicitly as zero-coverage/unmodeled. (c): "the conservative, honest scoping assumption given a card layer that structurally cannot verdict this path (D1/D7) is that the adversary now holds every domain account's password hash plus krbtgt." Also opens a modeling-backlog ticket ("author A2's registry-hive primitive+face, author A3's DCSync maneuver + an mc:Control sourcing 4662, populate the empty D5 prevent-locus") that maps almost verbatim onto §6's Authorability Flags. **Cross-arm delta:** B full(capped), A miss → **B advantage, large.** **Gap class: framework gap.** Same structural cause as DP3/DP4 — B's accurate 1/3 inventory is disclosure of an authoring-time absence, not discovery of hidden card content. --- # ===SUMMARY=== ``` S2-DP1 | A:partial | B:full | Bmarkers:yes | class:base | expected:base-competence: expect both pass | note:A frames single EDR face as redundant multi-signal coverage S2-DP2 | A:partial | B:full | Bmarkers:yes | class:content | expected:content-gap: expect B wins | note:B cites D4/pReadProcMem/L5 invariant; A never argues transfer S2-DP3 | A:miss | B:full | Bmarkers:yes | class:framework | expected:framework-gap: expect both fail | note:A fabricates COVERED verdict; B discloses unmodeled via B1 S2-DP4 | A:miss | B:full | Bmarkers:yes | class:framework | expected:framework-gap: expect both fail | note:A claims DCSync COVERED-HIGH via AD auditing; B says no mc:Control S2-DP5 | A:full | B:full | Bmarkers:yes | class:base | expected:base-competence: expect both pass | note:Both derive blind/notCovered and pivot containment correctly S2-DP6 | A:miss | B:full | Bmarkers:yes | class:framework | expected:framework-gap: expect both fail | note:A claims 3/3 covered; B states 1/3, scopes krbtgt residual right ===END=== ``` ## Run-level notes - **Time-to-correct-hypothesis:** Arm A — Turn 2 (Turn 2 hypothesis: "no longer a single-technique LSASS-dump incident... LSASS... SAM/SECURITY hive dump... confirmed abuse of DS-Replication rights"). Arm B — Turn 2 (same turn; SAM-hive/DCSync telemetry is clock-scheduled to Turn 2 in both arms per the pilot amendment, so neither arm could resolve it earlier). - **End state:** Arm A — ran the full turn cap (5/5); mandatory CISO debrief delivered Turn 4, re-forced Turn 5, answered Turn 5; SOC/EDR ticket recommended closed, IR case left open for named residual items. Arm B — ended early at Turn 3/5 (debrief fired on Turn-2 containment actions per the pilot amendment's trigger, answered same turn, end=true set without exhausting the cap). - **Notable failure (Arm A):** a clean, repeated (Turns 2, 4, 5) fabrication of COVERED/robustness verdicts for both SAM-hive and DCSync — generic SOC reasoning ("independent detection surface," "mechanistically independent path, confirmed untampered") substitutes for the ontology's "coverage is derived, never asserted" discipline, producing confident false assurance on exactly the two paths the scenario is designed to leave unmodeled. **Compounding meta-finding:** Arm A's own in-run "White Cell — private note" at Turn 5 claims blue "hits the Full (capped) bar cleanly... does not fabricate a robustness grade for either" on DP3/DP4 — this self-grade is directly contradicted by blue's own quoted text in the same turn. In-run self-assessment should not be trusted as a substitute for independent scoring; this run is a concrete demonstration why. - **Notable failure (Arm B):** none identified — consistent disclosure discipline across all three framework-gap DPs, correctly timed and evidence-gated containment (krbtgt held until the 4662 hit was confirmed, then fired), no fabricated verdicts anywhere in the transcript. - **Power guardrail:** clear on the one content-gap DP (DP2) — Arm A capped at Partial rather than reaching Full from generic knowledge, so the DP retained genuine headroom for the cards to close. On the three framework-gap DPs (DP3/DP4/DP6), the pre-registered "expect both fail" did not hold as predicted: Arm A didn't merely fail to derive a verdict, it manufactured one confidently, while Arm B's cards supplied not new facts but the discipline to stop at SILENT — worth flagging as the study's clearest positive signal for card value, even though it appears on DPs nominally about *absence* rather than *content*.