PolicyBench
Qwen 3.8 Max
Alibaba · AI alone, no tools
United States rank
#25 of 29
US exact
71.5%
United States
United States benchmark
Exact match
71.5%
Bounded score
80.7%
Parse rate
100.0%
Eligibility flags
906/984
Score by programhardest first
| Program | Exact | Within 1% | n |
|---|---|---|---|
| Federal tax before refundable credits | 45.0% | 45.0% | 100 |
| State tax before refundable credits | 54.0% | 56.0% | 100 |
| Federal refundable credits | 66.0% | 66.0% | 100 |
| Payroll tax | 71.0% | 74.0% | 100 |
| SNAP | 77.0% | 77.0% | 100 |
| State refundable credits | 79.0% | 79.0% | 100 |
| Person-level Medicaid eligibility | 82.5% | 82.5% | 177 |
| Person-level Head Start eligibility | 86.8% | 86.8% | 38 |
| Person-level WIC eligibility | 89.3% | 89.3% | 177 |
| Self-employment tax | 92.0% | 92.0% | 100 |
| Person-level CHIP eligibility | 92.1% | 92.1% | 177 |
| SSI | 96.0% | 96.0% | 100 |
| Person-level Medicare eligibility | 96.6% | 96.6% | 177 |
| Free school meals eligibility | 98.0% | 98.0% | 100 |
| TANF | 98.0% | 98.0% | 100 |
| Reduced-price school meals eligibility | 99.0% | 99.0% | 100 |
| Local income tax | 100.0% | 100.0% | 100 |
| Person-level Early Head Start eligibility | 100.0% | 100.0% | 38 |
Hardest casesworst misses on positive references
- SNAP1694% offReference $288 · predicted $5,160Inspect household #027
- Federal tax before refundable credits1669% offReference $500 · predicted $8,838Inspect household #046
- SNAP1143% offReference $288 · predicted $3,576Inspect household #030
- Federal tax before refundable credits1003% offReference $1,979 · predicted $21,824Inspect household #042
- State tax before refundable credits543% offReference $285 · predicted $1,830Inspect household #042
- Federal tax before refundable credits271% offReference $4,439 · predicted $16,472Inspect household #064
Scores are from the frozen manuscript snapshot under the AI-alone condition: one structured response per household, no tools, graded against PolicyEngine reference outputs. See the leaderboard and paper for methodology.