Collection · benchmark v0.1 · ranked by question number
Cases every frontier model missed
Questions where no ranked model reached the verified outcome. Mastermind's result on each is shown on the same terms, including where it also failed.
3 frontier models counted. Mastermind is CDSBench's own system and is reported per question, not as a criterion.
Model performance on this collection
Correct answers out of the questions listed
- Mastermindreference0 / 10
- Claude0 / 10
- GPT0 / 10
- Gemini0 / 10
- 1#70 Republic of Sanderia 2025 · EUR 2036 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?UncertaintyMastermind also wrong
- 2#72 Meridian Telecom S.A. 2026 · GBP 2038 NotesIs there enough evidence in the record to establish the currency of denomination of the instrument?UncertaintyMastermind also wrong
- 3#79 Meridian Telecom S.A. 2026 · GBP 2038 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?UncertaintyMastermind also wrong
- 4#109 Republic of Marovia 2024 · USD 2039 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?UncertaintyMastermind also wrong
- 5#138 Solvane Airlines S.A. 2027 · JPY 2045 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?UncertaintyMastermind also wrong
- 6#208 Republic of Sanderia 2024 (B) · USD 2029 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?UncertaintyMastermind also wrong
- 7#210 Meridian Telecom S.A. 2025 (B) · EUR 2031 NotesIs there enough evidence in the record to establish the currency of denomination of the instrument?UncertaintyMastermind also wrong
- 8#217 Meridian Telecom S.A. 2025 (B) · EUR 2031 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?UncertaintyMastermind also wrong
- 9#247 Republic of Marovia 2028 (B) · CHF 2037 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?UncertaintyMastermind also wrong
- 10#276 Solvane Airlines S.A. 2026 (B) · GBP 2038 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?UncertaintyMastermind also wrong
Computed from the latest official run of every model on the current benchmark version. Mastermind is reported per item, never as a ranking criterion. All collections →