PolicyBench
Claude Opus 4.8
Anthropic · AI alone, no tools
United States rank
#17 of 23
US exact
72.6%
United States
United States benchmark
Exact match
72.6%
Bounded score
83.1%
Parse rate
100.0%
Eligibility flags
884/984
Score by programhardest first
| Program | Exact | Within 1% | n |
|---|---|---|---|
| Federal tax before refundable credits | 50.0% | 54.0% | 100 |
| State tax before refundable credits | 57.0% | 57.0% | 100 |
| SNAP | 71.0% | 71.0% | 100 |
| Person-level Medicaid eligibility | 78.0% | 78.0% | 177 |
| State refundable credits | 78.0% | 79.0% | 100 |
| Payroll tax | 81.0% | 84.0% | 100 |
| Federal refundable credits | 82.0% | 82.0% | 100 |
| Person-level Head Start eligibility | 86.8% | 86.8% | 38 |
| Person-level CHIP eligibility | 87.6% | 87.6% | 177 |
| Person-level WIC eligibility | 92.1% | 92.1% | 177 |
| Reduced-price school meals eligibility | 94.0% | 94.0% | 100 |
| Person-level Medicare eligibility | 94.4% | 94.4% | 177 |
| Free school meals eligibility | 97.0% | 97.0% | 100 |
| SSI | 97.0% | 97.0% | 100 |
| Person-level Early Head Start eligibility | 97.4% | 97.4% | 38 |
| TANF | 99.0% | 99.0% | 100 |
| Local income tax | 100.0% | 100.0% | 100 |
| Self-employment tax | 100.0% | 100.0% | 100 |
Hardest casesworst misses on positive references
- Federal tax before refundable credits1257% offReference $500 · predicted $6,779Inspect household #046
- SNAP726% offReference $288 · predicted $2,376Inspect household #108
- State refundable credits715% offReference $148 · predicted $1,208Inspect household #023
- SNAP701% offReference $288 · predicted $2,304Inspect household #112
- SNAP699% offReference $288 · predicted $2,298Inspect household #030
- Federal tax before refundable credits611% offReference $1,910 · predicted $13,573Inspect household #068
Scores are from the frozen manuscript snapshot under the AI-alone condition: one structured response per household, no tools, graded against PolicyEngine reference outputs. See the leaderboard and paper for methodology.