PolicyBench
Qwen 3.7 Max
Alibaba · AI alone, no tools
United States rank
#15 of 23
US exact
73.6%
United States
United States benchmark
Exact match
73.6%
Bounded score
85.0%
Parse rate
100.0%
Eligibility flags
930/984
Score by programhardest first
| Program | Exact | Within 1% | n |
|---|---|---|---|
| Federal tax before refundable credits | 39.0% | 41.0% | 100 |
| State tax before refundable credits | 46.0% | 46.0% | 100 |
| Payroll tax | 73.0% | 81.0% | 100 |
| Federal refundable credits | 76.0% | 77.0% | 100 |
| SNAP | 77.0% | 77.0% | 100 |
| State refundable credits | 78.0% | 78.0% | 100 |
| Person-level Medicaid eligibility | 87.6% | 87.6% | 177 |
| Person-level Head Start eligibility | 89.5% | 89.5% | 38 |
| Person-level CHIP eligibility | 90.4% | 90.4% | 177 |
| Self-employment tax | 96.0% | 96.0% | 100 |
| SSI | 96.0% | 96.0% | 100 |
| Person-level Medicare eligibility | 97.2% | 97.2% | 177 |
| Person-level Early Head Start eligibility | 97.4% | 97.4% | 38 |
| Reduced-price school meals eligibility | 98.0% | 98.0% | 100 |
| Person-level WIC eligibility | 98.9% | 98.9% | 177 |
| Free school meals eligibility | 99.0% | 99.0% | 100 |
| TANF | 99.0% | 99.0% | 100 |
| Local income tax | 100.0% | 100.0% | 100 |
Hardest casesworst misses on positive references
- SNAP2344% offReference $288 · predicted $7,032Inspect household #045
- SNAP1143% offReference $288 · predicted $3,576Inspect household #112
- SNAP795% offReference $288 · predicted $2,576Inspect household #030
- SNAP571% offReference $461 · predicted $3,096Inspect household #023
- State tax before refundable credits534% offReference $514 · predicted $3,260Inspect household #039
- Federal tax before refundable credits379% offReference $6,311 · predicted -$17,600Inspect household #014
Scores are from the frozen manuscript snapshot under the AI-alone condition: one structured response per household, no tools, graded against PolicyEngine reference outputs. See the leaderboard and paper for methodology.