Gemini · Gemini 2.5 Pro · benchmark v0.1
Gemini missed 30 verified deliverability questions
Wrong answers, abstentions the record did not justify, and correct answers with unsupported evidence. Every row has a permanent page that can be cited or forwarded.
| # | Question | Category | Verified | Gemini | Failure |
|---|---|---|---|---|---|
| 7 | Republic of Norland 2024 · USD 2029 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 17 | Halvard Steel AG 2025 · EUR 2058 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 27 | Kingdom of Vestmark 2026 · ZAR 2033 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 37 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 47 | Federal Republic of Aurelia 2028 · CHF 2037 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 57 | Brightwater Energy Corp 2024 · USD 2034 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 67 | Republic of Sanderia 2025 · EUR 2036 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Review | Incorrect · Eligible | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 76 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Insufficient information | Incorrect · Eligible | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 86 | United Provinces of Kestrel 2027 · JPY 2040 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Incorrect · Eligible | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 96 | Northgate Shipping Ltd 2028 · CHF 2042 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 106 | Republic of Marovia 2024 · USD 2039 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Review | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 116 | Calder Chemicals N.V. 2025 · EUR 2041 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 126 | Commonwealth of Tessaly 2026 · GBP 2043 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 135 | Solvane Airlines S.A. 2027 · JPY 2045 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Insufficient information | Incorrect · Eligible | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 145 | Republic of Norland 2028 (B) · CHF 2047 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 155 | Halvard Steel AG 2024 (B) · USD 2056 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 165 | Kingdom of Vestmark 2025 (B) · NOK 2046 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 175 | Orinoco Retail Holdings plc 2026 (B) · GBP 2048 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 185 | Federal Republic of Aurelia 2027 (B) · JPY 2050 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 195 | Brightwater Energy Corp 2028 (B) · CHF 2052 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 205 | Republic of Sanderia 2024 (B) · USD 2029 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Review | Incorrect · Eligible | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 214 | Meridian Telecom S.A. 2025 (B) · EUR 2031 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Insufficient information | Incorrect · Eligible | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 224 | United Provinces of Kestrel 2026 (B) · GBP 2033 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Incorrect · Eligible | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 234 | Northgate Shipping Ltd 2027 (B) · JPY 2035 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 244 | Republic of Marovia 2028 (B) · CHF 2037 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Review | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 254 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 264 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 273 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Insufficient information | Incorrect · Eligible | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 283 | Republic of Norland 2027 (C) · JPY 2040 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 293 | Halvard Steel AG 2028 (C) · CHF 2064 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
Categories on this page: Deliverability. Category pages explain what each one tests: deliverability.