Gemini · Gemini 2.5 Pro · benchmark v0.1
Gemini missed 238 verified questions
Wrong answers, abstentions the record did not justify, and correct answers with unsupported evidence. Every row has a permanent page that can be cited or forwarded.
| # | Question | Category | Verified | Gemini | Failure |
|---|---|---|---|---|---|
| 66 | Republic of Sanderia 2025 · EUR 2036 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 67 | Republic of Sanderia 2025 · EUR 2036 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Review | Incorrect · Eligible | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 69 | Republic of Sanderia 2025 · EUR 2036 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 70 | Republic of Sanderia 2025 · EUR 2036 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 71 | Meridian Telecom S.A. 2026 · GBP 2038 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2038-08-22 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 72 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Is there enough evidence in the record to establish the currency of denomination of the instrument? | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 73 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 74 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 75 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 76 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Insufficient information | Incorrect · Eligible | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 78 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 79 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 80 | United Provinces of Kestrel 2027 · JPY 2040 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2040-09-25 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 81 | United Provinces of Kestrel 2027 · JPY 2040 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | JPY | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 82 | United Provinces of Kestrel 2027 · JPY 2040 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 83 | United Provinces of Kestrel 2027 · JPY 2040 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 84 | United Provinces of Kestrel 2027 · JPY 2040 Notes Is the instrument subordinated? | Seniority | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 85 | United Provinces of Kestrel 2027 · JPY 2040 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 86 | United Provinces of Kestrel 2027 · JPY 2040 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Incorrect · Eligible | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 88 | United Provinces of Kestrel 2027 · JPY 2040 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 90 | Northgate Shipping Ltd 2028 · CHF 2042 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2042-10-01 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 91 | Northgate Shipping Ltd 2028 · CHF 2042 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | CHF | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 92 | Northgate Shipping Ltd 2028 · CHF 2042 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 93 | Northgate Shipping Ltd 2028 · CHF 2042 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 94 | Northgate Shipping Ltd 2028 · CHF 2042 Notes Is the instrument subordinated? | Seniority | No | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 95 | Northgate Shipping Ltd 2028 · CHF 2042 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 96 | Northgate Shipping Ltd 2028 · CHF 2042 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 98 | Northgate Shipping Ltd 2028 · CHF 2042 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 100 | Republic of Marovia 2024 · USD 2039 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2039-11-04 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 101 | Republic of Marovia 2024 · USD 2039 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | USD | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 102 | Republic of Marovia 2024 · USD 2039 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 103 | Republic of Marovia 2024 · USD 2039 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 104 | Republic of Marovia 2024 · USD 2039 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 105 | Republic of Marovia 2024 · USD 2039 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 106 | Republic of Marovia 2024 · USD 2039 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Review | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 108 | Republic of Marovia 2024 · USD 2039 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 109 | Republic of Marovia 2024 · USD 2039 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 110 | Calder Chemicals N.V. 2025 · EUR 2041 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2041-12-07 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 111 | Calder Chemicals N.V. 2025 · EUR 2041 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | EUR | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 112 | Calder Chemicals N.V. 2025 · EUR 2041 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 113 | Calder Chemicals N.V. 2025 · EUR 2041 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 114 | Calder Chemicals N.V. 2025 · EUR 2041 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 115 | Calder Chemicals N.V. 2025 · EUR 2041 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 116 | Calder Chemicals N.V. 2025 · EUR 2041 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 118 | Calder Chemicals N.V. 2025 · EUR 2041 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 120 | Commonwealth of Tessaly 2026 · GBP 2043 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2043-01-10 | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 121 | Commonwealth of Tessaly 2026 · GBP 2043 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | GBP | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 122 | Commonwealth of Tessaly 2026 · GBP 2043 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 123 | Commonwealth of Tessaly 2026 · GBP 2043 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 124 | Commonwealth of Tessaly 2026 · GBP 2043 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
Categories on this page: Transferability, Deliverability, Evidence, Uncertainty, Fact extraction, Maturity, Seniority, Currency. Category pages explain what each one tests: transferability, deliverability, evidence.