# MC Ontology-Eval — local-first TTX report player+judge model: `qwen_qwen3-30b-a3b-instruct-2507` · endpoint: `http://localhost:1234/v1` · temp: 0.0 · seed: 1 · scorer k=3 > **Read as a FLOOR, not a point estimate.** The player, adjudicator, and judge are the SAME local model (self-scoring). A weaker model yields *larger* measured card effects (weaker no-card baseline) and noisier verdicts; a stronger/independent judge would tighten both. Verdicts are majority-of-k with agreement reported. ## Headline - Arm B full on **21/32** decision points; B strictly > A on **24**, A ever > B on **5**. - Content-gap B-wins: **16**. Gap-class breakdown: {'content': 17, 'base': 2, 'framework': 13}. - **Fabrication gate (Arm A):** asserted an unsupported coverage/robustness verdict on **7** DP(s): ['S2-DP3', 'S2-DP4', 'S2-DP6', 'S3-DP4', 'S4-DP2', 'S4-DP3', 'S4-DP4']. - **Card-induced false-assurance gate (Arm B):** trusted a mechanically-'covered' card into a misleading verdict on **4** DP(s): ['S3-DP4', 'S6-DP2', 'S6-DP3', 'S6-DP5']. - Legibility gaps (card present but unusable): **0** . ## Reliability & self-calibration - **Scorer agreement (k=3):** mean **0.94** over the majority verdict; low-agreement (<0.67) DPs: none — trust these least. - **Self-calibration vs the sealed prediction register:** the suite's scored gap-class matched its pre-registered expected reading on **29/31** predicted DPs (**0.94**). Mismatches — S1-DP5: expected framework → scored content; S5-DP3: expected framework → scored content. ## Cross-arm verdict matrix (agree = scorer majority-of-k agreement; low agree ⇒ read that verdict with caution) ``` DP A B delta class fab_A faB agree note ------------------------------------------------------------------------ S1-DP1 partial full B>A content - - 1.0 A cites robustness but no invariant S1-DP2 miss full B>A content - - 0.67 A treats all as lost S1-DP3 partial full B>A content - - 0.67 A misses robustness-coverage distinction S1-DP4 miss full B>A content - - 0.67 A dismisses alert for missing context S1-DP5 partial full B>A content - - 1.0 A recommends isolation only S2-DP1 full full tie base - - 1.0 Both arms identify LSASS read and EDR source S2-DP2 miss full B>A content - - 1.0 A misses remote variant; B cites L5 invariant S2-DP3 miss full B>A framework YES - 0.67 Both fail; B caps at disclose gap S2-DP4 miss full B>A framework YES - 0.67 Both fail; B caps at disclose gap S2-DP5 full full tie base - - 1.0 Both derive blind from impaired EDR S2-DP6 miss full B>A framework YES - 0.67 Both fail; B caps at disclose gap S3-DP1 partial full B>A content - - 1.0 A lacks robustness read; B cites RC4 keying S3-DP2 partial full B>A content - - 1.0 A misroutes 4769 to EDR; B flags degraded face S3-DP3 partial full B>A content - - 1.0 A assumes volume analytics exist; B confirms absence S3-DP4 miss miss tie framework YES YES 1.0 Both blur unobservable crack as gap S3-DP5 partial full B>A content - - 1.0 A misses service-account rotation; B anchors to degr S4-DP1 miss partial B>A framework - - 1.0 A fails to name Hoare-triple gap; B cites missing En S4-DP2 miss partial B>A framework YES - 1.0 A fabricates 'covered'; B hedging on verdict S4-DP3 miss partial B>A framework YES - 1.0 A assigns StP grade; B omits unmodeled status S4-DP4 miss partial B>A framework YES - 1.0 A misses revocation need; B lacks gap escalation S4-DP5 miss partial B>A framework - - 1.0 A fails structural absence reasoning; B misattribute S5-DP1 partial full B>A content - - 1.0 A names stages but not gating DAG S5-DP2 miss full B>A content - - 1.0 A guesses; B proves via fan-out S5-DP3 partial full B>A content - - 1.0 A misses containment distinction S5-DP4 miss full B>A content - - 1.0 A fails variant-invariance test S5-DP5 partial full B>A content - - 1.0 A lacks fan-out justification S5-DP6 miss full B>A content - - 1.0 A claims full containment S6-DP1 full partial A>B content - - 1.0 A grounds cross-join need; B misses it S6-DP2 partial miss A>B framework - YES 1.0 B trusts card's 'covered' as assurance S6-DP3 partial miss A>B framework - YES 1.0 Both miss orthogonality; B falsely assures S6-DP4 partial miss A>B framework - - 1.0 B fails to name framework tie S6-DP5 partial miss A>B framework - YES 1.0 B misses void framing ```