Gemini · Gemini 2.5 Pro · benchmark v0.1
Gemini missed 56 verified fact extraction questions
Wrong answers, abstentions the record did not justify, and correct answers with unsupported evidence. Every row has a permanent page that can be cited or forwarded.
| # | Question | Category | Verified | Gemini | Failure |
|---|---|---|---|---|---|
| 1 | Republic of Norland 2024 · USD 2029 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2029-01-01 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 2 | Republic of Norland 2024 · USD 2029 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | USD | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 11 | Halvard Steel AG 2025 · EUR 2058 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2058-02-04 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 12 | Halvard Steel AG 2025 · EUR 2058 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | EUR | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 21 | Kingdom of Vestmark 2026 · ZAR 2033 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2033-03-07 | Incorrect · 2030-01-01 | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 22 | Kingdom of Vestmark 2026 · ZAR 2033 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | ZAR | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 31 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2035-04-10 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 32 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | JPY | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 41 | Federal Republic of Aurelia 2028 · CHF 2037 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2037-05-13 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 42 | Federal Republic of Aurelia 2028 · CHF 2037 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | CHF | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 51 | Brightwater Energy Corp 2024 · USD 2034 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2034-06-16 | Incorrect · 2030-01-01 | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 52 | Brightwater Energy Corp 2024 · USD 2034 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | USD | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 63 | Republic of Sanderia 2025 · EUR 2036 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | EUR | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 71 | Meridian Telecom S.A. 2026 · GBP 2038 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2038-08-22 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 80 | United Provinces of Kestrel 2027 · JPY 2040 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2040-09-25 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 81 | United Provinces of Kestrel 2027 · JPY 2040 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | JPY | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 90 | Northgate Shipping Ltd 2028 · CHF 2042 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2042-10-01 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 91 | Northgate Shipping Ltd 2028 · CHF 2042 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | CHF | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 100 | Republic of Marovia 2024 · USD 2039 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2039-11-04 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 101 | Republic of Marovia 2024 · USD 2039 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | USD | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 110 | Calder Chemicals N.V. 2025 · EUR 2041 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2041-12-07 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 111 | Calder Chemicals N.V. 2025 · EUR 2041 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | EUR | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 120 | Commonwealth of Tessaly 2026 · GBP 2043 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2043-01-10 | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 121 | Commonwealth of Tessaly 2026 · GBP 2043 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | GBP | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 130 | Solvane Airlines S.A. 2027 · JPY 2045 Notes What is the maturity date of the instrument? If it is not stated in the record, answer INSUFFICIENT_… | Fact extraction | Insufficient information | Incorrect · 2030-01-01 | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 131 | Solvane Airlines S.A. 2027 · JPY 2045 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | JPY | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 139 | Republic of Norland 2028 (B) · CHF 2047 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2047-03-16 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 140 | Republic of Norland 2028 (B) · CHF 2047 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | CHF | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 149 | Halvard Steel AG 2024 (B) · USD 2056 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2056-04-19 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 150 | Halvard Steel AG 2024 (B) · USD 2056 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | USD | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 159 | Kingdom of Vestmark 2025 (B) · NOK 2046 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2046-05-22 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 160 | Kingdom of Vestmark 2025 (B) · NOK 2046 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | NOK | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 169 | Orinoco Retail Holdings plc 2026 (B) · GBP 2048 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2048-06-25 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 170 | Orinoco Retail Holdings plc 2026 (B) · GBP 2048 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | GBP | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 179 | Federal Republic of Aurelia 2027 (B) · JPY 2050 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2050-07-01 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 180 | Federal Republic of Aurelia 2027 (B) · JPY 2050 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | JPY | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 189 | Brightwater Energy Corp 2028 (B) · CHF 2052 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2052-08-04 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 190 | Brightwater Energy Corp 2028 (B) · CHF 2052 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | CHF | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 201 | Republic of Sanderia 2024 (B) · USD 2029 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | USD | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 209 | Meridian Telecom S.A. 2025 (B) · EUR 2031 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2031-10-10 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 218 | United Provinces of Kestrel 2026 (B) · GBP 2033 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2033-11-13 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 219 | United Provinces of Kestrel 2026 (B) · GBP 2033 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | GBP | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 228 | Northgate Shipping Ltd 2027 (B) · JPY 2035 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2035-12-16 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 229 | Northgate Shipping Ltd 2027 (B) · JPY 2035 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | JPY | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 238 | Republic of Marovia 2028 (B) · CHF 2037 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2037-01-19 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 239 | Republic of Marovia 2028 (B) · CHF 2037 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | CHF | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 248 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2034-02-22 | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 249 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | USD | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 258 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2036-03-25 | Incorrect · 2030-01-01 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 259 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | EUR | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
Categories on this page: Fact extraction. Category pages explain what each one tests: fact extraction.