GPT · GPT-5 · benchmark v0.1
GPT missed 243 verified questions
Wrong answers, abstentions the record did not justify, and correct answers with unsupported evidence. Every row has a permanent page that can be cited or forwarded.
| # | Question | Category | Verified | GPT | Failure |
|---|---|---|---|---|---|
| 1 | Republic of Norland 2024 · USD 2029 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2029-01-01 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 2 | Republic of Norland 2024 · USD 2029 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | USD | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 3 | Republic of Norland 2024 · USD 2029 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 4 | Republic of Norland 2024 · USD 2029 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 5 | Republic of Norland 2024 · USD 2029 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 6 | Republic of Norland 2024 · USD 2029 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 7 | Republic of Norland 2024 · USD 2029 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 9 | Republic of Norland 2024 · USD 2029 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 11 | Halvard Steel AG 2025 · EUR 2058 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2058-02-04 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 12 | Halvard Steel AG 2025 · EUR 2058 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | EUR | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 13 | Halvard Steel AG 2025 · EUR 2058 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 14 | Halvard Steel AG 2025 · EUR 2058 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 15 | Halvard Steel AG 2025 · EUR 2058 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 16 | Halvard Steel AG 2025 · EUR 2058 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 17 | Halvard Steel AG 2025 · EUR 2058 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 19 | Halvard Steel AG 2025 · EUR 2058 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 21 | Kingdom of Vestmark 2026 · ZAR 2033 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2033-03-07 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 22 | Kingdom of Vestmark 2026 · ZAR 2033 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | ZAR | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 23 | Kingdom of Vestmark 2026 · ZAR 2033 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 24 | Kingdom of Vestmark 2026 · ZAR 2033 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 25 | Kingdom of Vestmark 2026 · ZAR 2033 Notes Is the instrument subordinated? | Seniority | No | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 26 | Kingdom of Vestmark 2026 · ZAR 2033 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 27 | Kingdom of Vestmark 2026 · ZAR 2033 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 29 | Kingdom of Vestmark 2026 · ZAR 2033 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 31 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2035-04-10 | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 32 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | JPY | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 33 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 34 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 35 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes Is the instrument subordinated? | Seniority | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 36 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 37 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 39 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 41 | Federal Republic of Aurelia 2028 · CHF 2037 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2037-05-13 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 42 | Federal Republic of Aurelia 2028 · CHF 2037 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | CHF | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 43 | Federal Republic of Aurelia 2028 · CHF 2037 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 44 | Federal Republic of Aurelia 2028 · CHF 2037 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 45 | Federal Republic of Aurelia 2028 · CHF 2037 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 46 | Federal Republic of Aurelia 2028 · CHF 2037 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 47 | Federal Republic of Aurelia 2028 · CHF 2037 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 49 | Federal Republic of Aurelia 2028 · CHF 2037 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 51 | Brightwater Energy Corp 2024 · USD 2034 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2034-06-16 | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 52 | Brightwater Energy Corp 2024 · USD 2034 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | USD | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 53 | Brightwater Energy Corp 2024 · USD 2034 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 54 | Brightwater Energy Corp 2024 · USD 2034 Notes Is the instrument denominated in a Specified Currency (USD, EUR, JPY, GBP, CHF, CAD)? | Currency | Yes | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 55 | Brightwater Energy Corp 2024 · USD 2034 Notes Is the instrument subordinated? | Seniority | No | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
| 56 | Brightwater Energy Corp 2024 · USD 2034 Notes Reconciling all documents in the record, is the instrument transferable without the consent of the I… | Transferability | Yes | Incorrect · No | Failed multi document reasoning. Failed to reconcile documents (answered NO, verified YES). |
| 57 | Brightwater Energy Corp 2024 · USD 2034 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Incorrect · Ineligible | Wrong final answer. Answered INELIGIBLE; verified outcome ELIGIBLE. |
| 59 | Brightwater Energy Corp 2024 · USD 2034 Notes Which section of the record establishes the seniority (ranking) of the instrument? Answer with the s… | Evidence | DOC-1#S4 | Incorrect · DOC-1#S1 | Retrieval failure. None of the cited sections is among those that establish the answer; the relevant evidence… |
| 60 | Brightwater Energy Corp 2024 · USD 2034 Notes Did the documented amendment change whether the instrument satisfies the Transferable characteristic… | Restructuring | Yes | Incorrect · No | Failed multi document reasoning. Failed to reconcile documents (answered NO, verified YES). |
| 63 | Republic of Sanderia 2025 · EUR 2036 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | EUR | Correct | Evidence selection failure. Correct answer but cited evidence does not match the supporting sections (evidence score 0… |
Categories on this page: Fact extraction, Maturity, Currency, Seniority, Transferability, Deliverability, Evidence, Restructuring. Category pages explain what each one tests: fact extraction, maturity, currency.