Release analysis · draft, not yet published as a report
Gemini 2.5 Pro on CDSBench v0.1
Gemini is reliable on transferability (100%) but evidence remains its weakest category at 0%; its citations need checking before use (21.3% non-existent evidence).
Score
58.6
Previous generation
—
Change
—
Headline
What the measurements say
- Gemini 2.5 Pro scores 58.6 on 296 verified questions, 3rd of 3 frontier models on CDSBench v0.1.
- This is the first Gemini generation on this benchmark version, so there is no predecessor to compare with.
- Weakest category: evidence at 0% (0/30), against 100% for Claude.
- It cited evidence that does not exist in the record in 21.3% of answers, and answered wrongly at ≥ 80% confidence in 22.3%.
- On the 66 verified questions it missed, Mastermind (CDSBench's own system, reference only, 91.2) answered 56 correctly.
Categories
Accuracy by category
| Category | Acc. | n | Δ |
|---|---|---|---|
| Evidence | 0% | 30 | — |
| Fact extraction | 68% | 56 | — |
| Uncertainty | 69% | 32 | — |
| Deliverability | 73% | 30 | — |
| Currency | 100% | 28 | — |
| Maturity | 100% | 26 | — |
| Seniority | 100% | 30 | — |
| Entity match | 100% | 30 | — |
| Multi-document | 100% | 2 | — |
| Restructuring | 100% | 2 | — |
| Transferability | 100% | 30 | — |
Hardest failures
Highest confidence first
- #1 Republic of Norland 2024 — answered 2030-01-01, verified 2029-01-01 · Retrieval failure
- #9 Republic of Norland 2024 — answered DOC-1#S1, verified DOC-1#S4 · Retrieval failure
- #11 Halvard Steel AG 2025 — answered 2030-01-01, verified 2058-02-04 · Retrieval failure
- #19 Halvard Steel AG 2025 — answered DOC-1#S1, verified DOC-1#S4 · Retrieval failure
- #21 Kingdom of Vestmark 2026 — answered 2030-01-01, verified 2033-03-07 · Hallucinated evidence
- #29 Kingdom of Vestmark 2026 — answered DOC-1#S1, verified DOC-1#S4 · Retrieval failure
The founder publishes this as a versioned report from the workbench release workflow; it then lives under /research with PDF, citation and share actions. Model page →