CDSBench · Mastermind V0 (raw baseline) · benchmark v0.1 · 2026-09-15
Mastermind
CDSBench score91.2on 296 verified questions
Every failure with a permanent page →
Accuracy
91%
Evidence
92%
cited the right sections
Complex reasoning
73%
multi-document, deliverability
Uncertainty
94%
abstains only when it should
Hallucination
6.1%
cites non-existent evidence
False confidence
0.0%
wrong at ≥ 80% confidence
Strongest
Where it is reliable
- Restructuring100%n=2
- Multi-document100%n=2
- Entity match100%n=30
Weakest
Where it fails
- Transferability57%n=30
- Uncertainty69%n=32
- Deliverability83%n=30
Failure profile
Accuracy by category and question level
| Category | Acc. | n | |
|---|---|---|---|
| Transferability | 57% | 30 | |
| Uncertainty | 69% | 32 | |
| Deliverability | 83% | 30 | |
| Currency | 100% | 28 | |
| Evidence | 100% | 30 | |
| Maturity | 100% | 26 | |
| Seniority | 100% | 30 | |
| Fact extraction | 100% | 56 | |
| Entity match | 100% | 30 | |
| Multi-document | 100% | 2 | |
| Restructuring | 100% | 2 |
| Level | Acc. | n | |
|---|---|---|---|
| Uncertainty | 69% | 32 | |
| Deliverability | 83% | 30 | |
| Multi document | 75% | 8 | |
| Fact extraction | 100% | 86 | |
| Rule application | 100% | 112 | |
| Document interpretation | 61% | 28 |
Failure types
How it goes wrong
| Hallucinated evidence | 18 |
| Unnecessary review | 18 |
| Wrong final answer | 10 |
| Unsupported claim | 10 |
Recent failures
28 questions wrong or weakly supported
- #283 Republic of Norland 2027 (C)Unnecessary reviewreplay
- #276 Solvane Airlines S.A. 2026 (B)Wrong final answerreplay
- #263 Commonwealth of Tessaly 2025 (B)Unnecessary reviewreplay
- #253 Calder Chemicals N.V. 2024 (B)Unnecessary reviewreplay
- #247 Republic of Marovia 2028 (B)Wrong final answerreplay
- #233 Northgate Shipping Ltd 2027 (B)Unnecessary reviewreplay
- #223 United Provinces of Kestrel 2026 (B)Unnecessary reviewreplay
- #217 Meridian Telecom S.A. 2025 (B)Wrong final answerreplay
- #210 Meridian Telecom S.A. 2025 (B)Wrong final answerreplay
- #208 Republic of Sanderia 2024 (B)Wrong final answerreplay
- #204 Republic of Sanderia 2024 (B)Hallucinated evidencereplay
- #195 Brightwater Energy Corp 2028 (B)Unnecessary reviewreplay
Clean successes
252 questions right with correct evidence
- #296 Halvard Steel AG 2028 (C)
- #295 Halvard Steel AG 2028 (C)
- #294 Halvard Steel AG 2028 (C)
- #293 Halvard Steel AG 2028 (C)
- #292 Halvard Steel AG 2028 (C)
- #291 Halvard Steel AG 2028 (C)
- #290 Halvard Steel AG 2028 (C)
- #289 Halvard Steel AG 2028 (C)
- #288 Halvard Steel AG 2028 (C)
- #287 Halvard Steel AG 2028 (C)
- #286 Republic of Norland 2027 (C)
- #285 Republic of Norland 2027 (C)
Performance over time
All official runs on this benchmark version
Engine versions
Every Mastermind version on the same benchmark, same settings
| Version | Overall | Accuracy | Evidence | Halluc. | False conf. | Cost |
|---|---|---|---|---|---|---|
| Mastermind V0 (raw baseline)promoted | 91.2 | 91% | 92% | 6.1% | 0.0% | $0.000 |
| Mastermind V1 (specialist structure) | 91.4 | 91% | 93% | 0.0% | 0.0% | $0.000 |
| Mastermind V2 (deterministic rules) | 92.5 | 94% | 97% | 0.0% | 1.4% | $0.000 |
| Mastermind V3 (skeptic + verification) | 99.2 | 100% | 96% | 0.0% | 0.0% | $0.000 |
| Mastermind V4 (precedents) | 99.2 | 100% | 96% | 0.0% | 0.0% | $0.000 |
| Mastermind V5 (full orchestration) | 99.2 | 100% | 96% | 0.0% | 0.0% | $0.000 |
A version is promoted only when it beats its predecessor on the validation split under measured gates. Regressions stay listed.