Release analysis · draft, not yet published as a report
Mastermind V2 (deterministic rules) on CDSBench v0.1
Mastermind is reliable on fact extraction (100%) but transferability remains its weakest category at 57%; the generation improved on uncertainty while regressing on entity match.
Score
92.5
Previous generation
91.4
Mastermind V1 (specialist structure)
Change
+1.1
Headline
What the measurements say
- Mastermind V2 (deterministic rules) scores 92.5 on 296 verified questions on CDSBench v0.1.
- Against Mastermind V1 (specialist structure) (91.4), overall moved up 1.1 points; strongest gain in uncertainty (+25 pts); largest regression in entity match (−7 pts).
- Weakest category: transferability at 57% (17/30), against 57% for Claude.
- It did not cite non-existent evidence on this run; wrong answers at ≥ 80% confidence: 1.4%.
Categories
Accuracy by category
| Category | Acc. | n | Δ |
|---|---|---|---|
| Transferability | 57% | 30 | ±0 |
| Entity match | 93% | 30 | −7 |
| Deliverability | 93% | 30 | +10 |
| Uncertainty | 94% | 32 | +25 |
| Currency | 100% | 28 | ±0 |
| Evidence | 100% | 30 | ±0 |
| Maturity | 100% | 26 | ±0 |
| Seniority | 100% | 30 | ±0 |
| Fact extraction | 100% | 56 | ±0 |
| Multi-document | 100% | 2 | ±0 |
| Restructuring | 100% | 2 | ±0 |
Hardest failures
Highest confidence first
- #67 Republic of Sanderia 2025 — answered Eligible, verified Review · Failed to notice contradiction
- #205 Republic of Sanderia 2024 (B) — answered Eligible, verified Review · Failed to notice contradiction
- #61 Brightwater Energy Corp 2024 — answered No, verified Yes · Wrong final answer
- #199 Brightwater Energy Corp 2028 (B) — answered No, verified Yes · Wrong final answer
- #56 Brightwater Energy Corp 2024 — answered Review, verified Yes · Unnecessary review
- #117 Calder Chemicals N.V. 2025 — answered Review, verified Yes · Unnecessary review
The founder publishes this as a versioned report from the workbench release workflow; it then lives under /research with PDF, citation and share actions. Model page →