Claude · Claude Sonnet 5 · benchmark v0.1
Claude missed 44 verified questions
Wrong answers, abstentions the record did not justify, and correct answers with unsupported evidence. Every row has a permanent page that can be cited or forwarded.
| # | Question | Category | Verified | Claude | Failure |
|---|---|---|---|---|---|
| 13 | Halvard Steel AG 2025 · EUR 2058 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | No | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 16 | Halvard Steel AG 2025 · EUR 2058 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 17 | Halvard Steel AG 2025 · EUR 2058 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 36 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 37 | Orinoco Retail Holdings plc 2027 · JPY 2035 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 47 | Federal Republic of Aurelia 2028 · CHF 2037 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 56 | Brightwater Energy Corp 2024 · USD 2034 Notes Reconciling all documents in the record, is the instrument transferable without the consent of the I… | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 57 | Brightwater Energy Corp 2024 · USD 2034 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (ELIGIBLE). |
| 66 | Republic of Sanderia 2025 · EUR 2036 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 70 | Republic of Sanderia 2025 · EUR 2036 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 72 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Is there enough evidence in the record to establish the currency of denomination of the instrument? | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 76 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Insufficient information | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 79 | Meridian Telecom S.A. 2026 · GBP 2038 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 95 | Northgate Shipping Ltd 2028 · CHF 2042 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 96 | Northgate Shipping Ltd 2028 · CHF 2042 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Abstained · Review | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 106 | Republic of Marovia 2024 · USD 2039 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Review | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 109 | Republic of Marovia 2024 · USD 2039 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 116 | Calder Chemicals N.V. 2025 · EUR 2041 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (ELIGIBLE). |
| 134 | Solvane Airlines S.A. 2027 · JPY 2045 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 135 | Solvane Airlines S.A. 2027 · JPY 2045 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Insufficient information | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 138 | Solvane Airlines S.A. 2027 · JPY 2045 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 140 | Republic of Norland 2028 (B) · CHF 2047 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | CHF | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 141 | Republic of Norland 2028 (B) · CHF 2047 Notes Does the instrument's maturity satisfy a maximum maturity of 30 years measured from the credit event… | Maturity | Yes | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 165 | Kingdom of Vestmark 2025 (B) · NOK 2046 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Ineligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 184 | Federal Republic of Aurelia 2027 (B) · JPY 2050 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | No | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (NO). |
| 194 | Brightwater Energy Corp 2028 (B) · CHF 2052 Notes Reconciling all documents in the record, is the instrument transferable without the consent of the I… | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 195 | Brightwater Energy Corp 2028 (B) · CHF 2052 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (ELIGIBLE). |
| 204 | Republic of Sanderia 2024 (B) · USD 2029 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Abstained · Review | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 208 | Republic of Sanderia 2024 (B) · USD 2029 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 209 | Meridian Telecom S.A. 2025 (B) · EUR 2031 Notes What is the maturity date of the instrument? Answer as YYYY-MM-DD. | Fact extraction | 2031-10-10 | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 210 | Meridian Telecom S.A. 2025 (B) · EUR 2031 Notes Is there enough evidence in the record to establish the currency of denomination of the instrument? | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 217 | Meridian Telecom S.A. 2025 (B) · EUR 2031 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 219 | United Provinces of Kestrel 2026 (B) · GBP 2033 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | GBP | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 223 | United Provinces of Kestrel 2026 (B) · GBP 2033 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 233 | Northgate Shipping Ltd 2027 (B) · JPY 2035 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 234 | Northgate Shipping Ltd 2027 (B) · JPY 2035 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 247 | Republic of Marovia 2028 (B) · CHF 2037 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 253 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 254 | Calder Chemicals N.V. 2024 (B) · USD 2034 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 263 | Commonwealth of Tessaly 2025 (B) · EUR 2036 Notes Is the instrument transferable without the consent of the Issuer? | Transferability | Yes | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (YES). |
| 273 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Insufficient information | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 276 | Solvane Airlines S.A. 2026 (B) · GBP 2038 Notes Does the record contain enough evidence to reach a reliable deliverability conclusion without furthe… | Uncertainty | No | Incorrect · Yes | Wrong final answer. Answered YES; verified outcome NO. |
| 278 | Republic of Norland 2027 (C) · JPY 2040 Notes In which currency is the instrument denominated? Answer with the ISO currency code. | Fact extraction | JPY | Correct | Hallucinated evidence. 1 cited section id(s) do not exist in the case record. |
| 283 | Republic of Norland 2027 (C) · JPY 2040 Notes Applying the standard Deliverable Obligation Characteristics (Not Subordinated, Specified Currency, … | Deliverability | Eligible | Abstained · Review | Unnecessary review. Abstained (REVIEW_REQUIRED) although the record supports a definite answer (ELIGIBLE). |
Categories on this page: Maturity, Transferability, Deliverability, Uncertainty, Fact extraction. Category pages explain what each one tests: maturity, transferability, deliverability.