PolicyBench
Claude Sonnet 5
Anthropic · AI alone, no tools
United States rank
#21 of 23
US exact
69.4%
United States
United States benchmark
Exact match
69.4%
Bounded score
80.7%
Parse rate
100.0%
Eligibility flags
903/984
Score by programhardest first
| Program | Exact | Within 1% | n |
|---|---|---|---|
| Federal tax before refundable credits | 44.0% | 47.0% | 100 |
| State tax before refundable credits | 54.0% | 56.0% | 100 |
| SNAP | 69.0% | 69.0% | 100 |
| Person-level Medicaid eligibility | 74.6% | 74.6% | 177 |
| Payroll tax | 76.0% | 86.0% | 100 |
| Federal refundable credits | 78.0% | 78.0% | 100 |
| State refundable credits | 78.0% | 78.0% | 100 |
| Person-level CHIP eligibility | 91.5% | 91.5% | 177 |
| Person-level WIC eligibility | 94.4% | 94.4% | 177 |
| Person-level Head Start eligibility | 94.7% | 94.7% | 38 |
| Self-employment tax | 96.0% | 96.0% | 100 |
| Person-level Medicare eligibility | 96.0% | 96.0% | 177 |
| SSI | 97.0% | 97.0% | 100 |
| Free school meals eligibility | 99.0% | 99.0% | 100 |
| Local income tax | 99.0% | 99.0% | 100 |
| Reduced-price school meals eligibility | 99.0% | 99.0% | 100 |
| TANF | 99.0% | 99.0% | 100 |
| Person-level Early Head Start eligibility | 100.0% | 100.0% | 38 |
Hardest casesworst misses on positive references
- State refundable credits6216% offReference $19 · predicted $1,200Inspect household #043
- Federal tax before refundable credits2341% offReference $500 · predicted $12,200Inspect household #046
- SNAP1360% offReference $288 · predicted $4,200Inspect household #030
- SNAP584% offReference $288 · predicted $1,968Inspect household #108
- State tax before refundable credits561% offReference $514 · predicted $3,400Inspect household #039
- Federal refundable credits488% offReference $40 · predicted $234Inspect household #066
Scores are from the frozen manuscript snapshot under the AI-alone condition: one structured response per household, no tools, graded against PolicyEngine reference outputs. See the leaderboard and paper for methodology.