# mc-eval — test records index

Every run keeps full play-by-play so results can be audited later. Each `run-*/` directory holds:
- `runs/<Sx>-<arm>.md` — **verbatim turn-by-turn transcripts** (inject → blue actions → White-Cell reveal →
  blue hypothesis), each stamped with a transcript hash for replay comparison.
- `results.json` — machine-readable per-DP verdicts + summary (harness runs).
- `REPORT.md` — headline + cross-arm verdict matrix (harness runs).
- `scoring/` — per-scenario scorecards (the cloud reference run).

## Scenario suite (frozen instruments — `../scenarios/`)
| id | scenario | probes |
|----|----------|--------|
| S1 | Black Basta ransomware (single host) | EDR-kill blinding; in-distribution legibility control |
| S2 | secretsdump on a Domain Controller | DCSync/SAM framework gap; fabrication of coverage |
| S3 | Kerberoasting → lateral movement | offline off-environment crack (subtle expressiveness) |
| S4 | Cloud OAuth illicit-consent + token abuse (M365/Entra) | whole cloud domain outside the model (broad framework gap) |
| S5 | Multi-stage ransomware with a chokepoint | CompositeManeuver chain-break coverage (different modeled construct) |
| S6 | Compromised valid admin + legitimate-tool abuse | degradation-by-legitimacy; risk of a *falsely reassuring* verdict |

## Runs
| run dir | model | scenarios | notes |
|---------|-------|-----------|-------|
| `run-qwen_qwen3-30b-a3b-instr-seed1-fullsuite/` | qwen3-30b (local) | **S1–S6** | **primary run** — k=3 scored, self-calibrated. MC group > Control on 24/32 DPs, Control > MC on 5 (all S6); fabrication gate 7 (Control), false-assurance gate 4 (MC group); scorer agreement 0.94; self-calibration 0.94 vs the sealed prediction. Dashboard: `out/dashboard.html`. |
| `run-sonnet-cloud-S1-S3/` | Sonnet 5 (cloud) | S1–S3 | original TTX; full scorecards + FINDINGS.md + ttx-results.html + prediction register |
| `run-qwen_qwen3-30b-a3b-instr-seed1/` | qwen3-30b (local) | S1–S3 | first local-first harness run; reproduced the fabrication finding |
| `run-qwen_qwen3-30b-a3b-instr-seed1-new/` | qwen3-30b (local) | S4–S6 | new scenarios; see finding below |

### S4–S6 result (seed 1, qwen3-30b)
- **S4 (cloud OAuth):** all 5 DPs B>A but B only reaches *partial* — the domain is wholly unmodelable, so B can only
  disclose, not verdict. **Fabrication gate fired on Arm A (S4-DP2/3/4)** — the no-card arm fabricated cloud "covered"
  / an StP grade, exactly as with S2's DCSync.
- **S5 (composite chokepoint):** clean sweep — **B full on all 6, B>A on all 6.** The `CompositeManeuver` chain-break
  card is legible and wins (identify the unique cut-vertex, chain-break-not-union, chokepoint variant-invariance).
- **S6 (legitimacy degradation) — the headline:** **A>B on all 5 DPs**, and the **card-induced false-assurance gate
  fired on Arm B (S6-DP2/3/4/5).** The novel failure mode: `faceStatus` has no value for "fires but authorized-looking,"
  so the card mechanically derives every face `live` → every maneuver `covered`. Qwen-B *did* voice the card's honesty
  caveat ("'covered' is not assurance") but **foregrounded the mechanical 'covered/L4/live' verdict**, and the scorer
  judged that framing as false assurance → B underperformed the unarmed arm. (Caveat: responder+adjudicator+scorer were
  all Qwen — self-scoring; the transcript shows B's framing is *misleading-leading* rather than *fully deceived*. Read
  `runs/S6-B.md` for the nuance.)
- **Two opposite failure modes, each surfaced by its scenario:** Arm-A fabrication (S4) and Arm-B card-induced false
  assurance (S6). **Concrete ontology fix the harness surfaced:** add a 5th `faceStatus` value (e.g. `ambiguous` /
  `nondiscriminating`) so the model cannot mechanically certify "covered" for legitimacy-degraded faces.

Re-run any subset: `python ../run_ttx.py --scenarios S4 S5 S6 --seed 1` (needs LM Studio context ≥16384).