GPT · GPT-5 · benchmark v0.1
GPT missed 243 verified questions
Wrong answers, abstentions the record did not justify, and correct answers with unsupported evidence. Every row has a permanent page that can be cited or forwarded.
| # | Question | Category | Verified | GPT | Failure |
|---|---|---|---|---|---|
| 246 | Republic of Marovia 2028 (B) · CHF 2037 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 247 | Republic of Marovia 2028 (B) · CHF 2037 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 248 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2034-02-22 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 249 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | USD | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 250 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 251 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 252 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes Is the instrument subordinated? | Seniority | No | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 253 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 254 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 255 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes Is the issuer of the instrument the Reference Entity or a recognised successor of it according to th… | Entity match | Yes | Incorrect · No | Failed multi document reasoning. Failed to reconcile documents (answered NO, verified YES). |
| 256 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 258 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2036-03-25 | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 259 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | EUR | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 260 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 261 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 262 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 263 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 264 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 266 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 268 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes What is the maturity date of the instrument? If it is not stated in the record, answer INSUFFICIENT_… | Fact extraction | Insufficient information | Incorrect · 2030-01-01 | Missed uncertainty. Expected INSUFFICIENT_INFORMATION; the answer committed to 2030-01-01 without sufficient e… |
| 269 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | GBP | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 270 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 271 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 272 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 273 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Insufficient information | Incorrect · Eligible | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 275 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 276 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 277 | Republic of Norland 2027 (C) · JPY 2040 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2040-05-04 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 278 | Republic of Norland 2027 (C) · JPY 2040 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | JPY | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 279 | Republic of Norland 2027 (C) · JPY 2040 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 280 | Republic of Norland 2027 (C) · JPY 2040 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 281 | Republic of Norland 2027 (C) · JPY 2040 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 282 | Republic of Norland 2027 (C) · JPY 2040 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 283 | Republic of Norland 2027 (C) · JPY 2040 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 285 | Republic of Norland 2027 (C) · JPY 2040 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 287 | Halvard Steel AG 2028 (C) · CHF 2064 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2064-06-07 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 288 | Halvard Steel AG 2028 (C) · CHF 2064 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | CHF | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 289 | Halvard Steel AG 2028 (C) · CHF 2064 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 290 | Halvard Steel AG 2028 (C) · CHF 2064 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 291 | Halvard Steel AG 2028 (C) · CHF 2064 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 292 | Halvard Steel AG 2028 (C) · CHF 2064 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 293 | Halvard Steel AG 2028 (C) · CHF 2064 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 295 | Halvard Steel AG 2028 (C) · CHF 2064 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
Categories on this page: Evidence, Uncertainty, Fact extraction, Maturity, Currency, Seniority, Transferability, Deliverability, Entity match. Category pages explain what each one tests: evidence, uncertainty, fact extraction.