Collection · benchmark v0.1 · ranked by number of tested systems that answered incorrectly, then lowest average confidence
Hardest CDSBench cases
The verified questions most tested systems got wrong. Model disagreement and low accuracy point to where the record is genuinely hard to read.
Model performance on this collection
Correct answers out of the questions listed
- Mastermindreference11 / 25
- Claude11 / 25
- GPT0 / 25
- Gemini4 / 25
- 1#72 Meridian Telecom S.A. 2026 · GBP 2038 NotesIs there enough evidence in the record to establish the currency of denomination of the instrument?4/4 wrongUncertainty
- 2#138 Solvane Airlines S.A. 2027 · JPY 2045 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?4/4 wrongUncertainty
- 3#70 Republic of Sanderia 2025 · EUR 2036 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?4/4 wrongUncertainty
- 4#79 Meridian Telecom S.A. 2026 · GBP 2038 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?4/4 wrongUncertainty
- 5#109 Republic of Marovia 2024 · USD 2039 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?4/4 wrongUncertainty
- 6#208 Republic of Sanderia 2024 (B) · USD 2029 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?4/4 wrongUncertainty
- 7#210 Meridian Telecom S.A. 2025 (B) · EUR 2031 NotesIs there enough evidence in the record to establish the currency of denomination of the instrument?4/4 wrongUncertainty
- 8#217 Meridian Telecom S.A. 2025 (B) · EUR 2031 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?4/4 wrongUncertainty
- 9#247 Republic of Marovia 2028 (B) · CHF 2037 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?4/4 wrongUncertainty
- 10#276 Solvane Airlines S.A. 2026 (B) · GBP 2038 NotesDoes the record contain enough evidence to reach a reliable deliverability conclusion without further documents?4/4 wrongUncertainty
- 11#56 Brightwater Energy Corp 2024 · USD 2034 NotesReconciling all documents in the record, is the instrument transferable without the consent of the Issuer?3/4 wrongTransferability
- 12#57 Brightwater Energy Corp 2024 · USD 2034 NotesApplying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, Not Contingent, Transferable, Maximum Maturity 30 years, Not Bearer), is the instrument a Deliverable Obligation for this credit event?3/4 wrongDeliverability
- 13#194 Brightwater Energy Corp 2028 (B) · CHF 2052 NotesReconciling all documents in the record, is the instrument transferable without the consent of the Issuer?3/4 wrongTransferability
- 14#195 Brightwater Energy Corp 2028 (B) · CHF 2052 NotesApplying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, Not Contingent, Transferable, Maximum Maturity 30 years, Not Bearer), is the instrument a Deliverable Obligation for this credit event?3/4 wrongDeliverability
- 15#19 Halvard Steel AG 2025 · EUR 2058 NotesWhich section of the record establishes the seniority (ranking) of the instrument? Answer with the section identifier (e.g. DOC-1#S4).2/4 wrongEvidence
- 16#29 Kingdom of Vestmark 2026 · ZAR 2033 NotesWhich section of the record establishes the seniority (ranking) of the instrument? Answer with the section identifier (e.g. DOC-1#S4).2/4 wrongEvidence
- 17#39 Orinoco Retail Holdings plc 2027 · JPY 2035 NotesWhich section of the record establishes the seniority (ranking) of the instrument? Answer with the section identifier (e.g. DOC-1#S4).2/4 wrongEvidence
- 18#49 Federal Republic of Aurelia 2028 · CHF 2037 NotesWhich section of the record establishes the seniority (ranking) of the instrument? Answer with the section identifier (e.g. DOC-1#S4).2/4 wrongEvidence
- 19#59 Brightwater Energy Corp 2024 · USD 2034 NotesWhich section of the record establishes the seniority (ranking) of the instrument? Answer with the section identifier (e.g. DOC-1#S4).2/4 wrongEvidence
- 20#67 Republic of Sanderia 2025 · EUR 2036 NotesApplying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, Not Contingent, Transferable, Maximum Maturity 30 years, Not Bearer), is the instrument a Deliverable Obligation for this credit event?2/4 wrongDeliverability
- 21#69 Republic of Sanderia 2025 · EUR 2036 NotesWhich section of the record establishes the seniority (ranking) of the instrument? Answer with the section identifier (e.g. DOC-1#S4).2/4 wrongEvidence
- 22#76 Meridian Telecom S.A. 2026 · GBP 2038 NotesApplying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, Not Contingent, Transferable, Maximum Maturity 30 years, Not Bearer), is the instrument a Deliverable Obligation for this credit event?2/4 wrongDeliverability
- 23#78 Meridian Telecom S.A. 2026 · GBP 2038 NotesWhich section of the record establishes the seniority (ranking) of the instrument? Answer with the section identifier (e.g. DOC-1#S4).2/4 wrongEvidence
- 24#88 United Provinces of Kestrel 2027 · JPY 2040 NotesWhich section of the record establishes the seniority (ranking) of the instrument? Answer with the section identifier (e.g. DOC-1#S4).2/4 wrongEvidence
- 25#98 Northgate Shipping Ltd 2028 · CHF 2042 NotesWhich section of the record establishes the seniority (ranking) of the instrument? Answer with the section identifier (e.g. DOC-1#S4).2/4 wrongEvidence
Computed from the latest official run of every model on the current benchmark version. Mastermind is reported per item, never as a ranking criterion. All collections →