PolicyBench leaderboard
Model rankings
23 models, ranked by exact-match score (United States)
Optionsscoring: exact · cases: all cases · weighting: household · all 18 programs
Filter restricts model scoring and the program breakdown table to the selected outputs. The scenario explorer remains unfiltered so each household's full prompt stays visible. Weights rescale to 100% over the active set.
Per-variable weights18 variables, sorted by Household
| Variable | Household | Aggregate | Equal |
|---|---|---|---|
| Person-level Medicaid eligibility | 29.86% | 16.02% | 5.56% |
| Federal tax before refundable credits | 19.14% | 42.18% | 5.56% |
| Payroll tax | 15.65% | 14.34% | 5.56% |
| Person-level Medicare eligibility | 10.74% | 5.42% | 5.56% |
| State tax before refundable credits | 6.00% | 10.37% | 5.56% |
| SNAP | 4.15% | 1.74% | 5.56% |
| Federal refundable credits | 3.70% | 2.13% | 5.56% |
| Person-level Early Head Start eligibility | 3.10% | 2.39% | 5.56% |
| Self-employment tax | 2.09% | 1.56% | 5.56% |
| SSI | 1.97% | 1.08% | 5.56% |
| Person-level Head Start eligibility | 1.18% | 0.84% | 5.56% |
| Free school meals eligibility | 0.74% | 0.57% | 5.56% |
| State refundable credits | 0.55% | 0.35% | 5.56% |
| Person-level WIC eligibility | 0.32% | 0.23% | 5.56% |
| Local income tax | 0.26% | 0.43% | 5.56% |
| TANF | 0.26% | 0.16% | 5.56% |
| Person-level CHIP eligibility | 0.18% | 0.12% | 5.56% |
| Reduced-price school meals eligibility | 0.10% | 0.05% | 5.56% |
Program breakdown
Bounded score by program and model (AI alone). Dollar targets use continuous relative-error partial credit; binary coverage flags use exact accuracy.
Program filterAll 18 programs
Shared with model scoring. The scenario explorer remains unfiltered so each household's full prompt stays visible. The table shows only selected outputs; model scores rescale selected weights to 100%.
| Program | Claude Fable 5 | Claude Opus 4.8 | Claude Opus 4.7 | Claude Sonnet 5 | Claude Sonnet 4.6 | Claude Haiku 4.5 | Grok 4.3 | Grok Build 0.1 | GPT-5.6 Sol | GPT-5.6 Terra | GPT-5.6 Luna | GPT-5.5 | GPT-5.4 mini | GPT-5.4 nano | Gemini 3.1 Pro Preview | Gemini 3.5 Flash | Gemini 3 Flash Preview | Gemini 3.1 Flash Lite Preview | DeepSeek V4 Pro | Kimi K2.6 | GLM-5.2 | MiniMax M3 | Qwen 3.7 Max | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Local income tax | 100% | 100% | 100% | 99% | 100% | 99% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 99% | 98% | 99% | 100% | 87% | 94% | 100% | 100% | 99% |
| TANF | 99% | 100% | 99% | 100% | 99% | 99% | 99% | 100% | 100% | 99% | 100% | 100% | 99% | 99% | 99% | 100% | 100% | 99% | 100% | 89% | 94% | 99% | 99% | 99% |
| Free school meals eligibility | 99% | 97% | 99% | 99% | 98% | 98% | 99% | 98% | 100% | 98% | 98% | 98% | 98% | 92% | 99% | 100% | 100% | 99% | 98% | 91% | 93% | 99% | 99% | 98% |
| Reduced-price school meals eligibility | 100% | 94% | 98% | 99% | 98% | 92% | 99% | 97% | 99% | 99% | 97% | 99% | 99% | 99% | 99% | 98% | 98% | 99% | 99% | 90% | 94% | 99% | 98% | 98% |
| Person-level Early Head Start eligibility | 100% | 97% | 100% | 100% | 95% | 87% | 97% | 100% | 100% | 100% | 100% | 100% | 92% | 97% | 100% | 100% | 100% | 97% | 100% | 95% | 89% | 97% | 97% | 97% |
| Person-level WIC eligibility | 99% | 92% | 95% | 94% | 99% | 93% | 98% | 100% | 100% | 99% | 100% | 100% | 86% | 96% | 99% | 100% | 99% | 99% | 99% | 91% | 93% | 97% | 99% | 97% |
| SSI | 98% | 98% | 98% | 98% | 99% | 97% | 97% | 97% | 97% | 98% | 98% | 96% | 97% | 97% | 97% | 98% | 98% | 97% | 98% | 87% | 91% | 96% | 98% | 97% |
| Self-employment tax | 99% | 100% | 100% | 97% | 99% | 93% | 99% | 99% | 100% | 98% | 99% | 100% | 96% | 96% | 100% | 100% | 99% | 98% | 99% | 60% | 90% | 96% | 96% | 96% |
| Person-level Head Start eligibility | 100% | 87% | 87% | 95% | 97% | 97% | 97% | 100% | 100% | 100% | 100% | 100% | 84% | 95% | 95% | 100% | 100% | 97% | 97% | 97% | 89% | 95% | 89% | 96% |
| Person-level Medicare eligibility | 89% | 94% | 90% | 96% | 94% | 97% | 98% | 98% | 94% | 99% | 97% | 90% | 96% | 91% | 97% | 97% | 97% | 97% | 98% | 90% | 89% | 97% | 97% | 95% |
| Person-level CHIP eligibility | 99% | 88% | 88% | 92% | 89% | 90% | 96% | 98% | 99% | 100% | 97% | 95% | 92% | 100% | 98% | 96% | 99% | 100% | 98% | 84% | 93% | 98% | 90% | 95% |
| Payroll tax | 98% | 98% | 99% | 98% | 100% | 91% | 97% | 97% | 99% | 99% | 98% | 99% | 82% | 63% | 94% | 93% | 95% | 98% | 95% | 48% | 88% | 84% | 97% | 92% |
| Person-level Medicaid eligibility | 94% | 78% | 86% | 75% | 87% | 82% | 89% | 94% | 97% | 91% | 94% | 95% | 84% | 71% | 97% | 97% | 95% | 90% | 94% | 82% | 86% | 84% | 88% | 88% |
| Federal refundable credits | 93% | 88% | 89% | 84% | 95% | 80% | 86% | 90% | 99% | 96% | 96% | 96% | 84% | 86% | 93% | 92% | 93% | 89% | 92% | 46% | 83% | 82% | 82% | 88% |
| SNAP | 86% | 78% | 81% | 75% | 84% | 81% | 79% | 86% | 89% | 88% | 87% | 87% | 77% | 80% | 87% | 90% | 88% | 84% | 87% | 79% | 84% | 82% | 83% | 84% |
| State refundable credits | 86% | 81% | 81% | 80% | 82% | 79% | 79% | 80% | 87% | 82% | 84% | 89% | 79% | 79% | 86% | 86% | 89% | 80% | 84% | 48% | 74% | 80% | 80% | 81% |
| State tax before refundable credits | 80% | 84% | 85% | 80% | 84% | 71% | 81% | 75% | 89% | 84% | 79% | 90% | 69% | 65% | 86% | 82% | 84% | 72% | 79% | 51% | 66% | 63% | 71% | 77% |
| Federal tax before refundable credits | 86% | 78% | 86% | 72% | 86% | 71% | 75% | 71% | 93% | 91% | 87% | 88% | 63% | 61% | 77% | 75% | 76% | 68% | 74% | 48% | 77% | 58% | 66% | 75% |
Scenario explorer
Inspect benchmark households, reference outputs, model answers, and the exact prompt sent to every model.
| Program | Reference | Fable 5 | Grok 4.3 | GPT-5.6 Sol | Pro Preview | V4 Pro | Kimi K2.6 | GLM-5.2 | MiniMax M3 | 3.7 Max |
|---|---|---|---|---|---|---|---|---|---|---|
| Federal tax before refundable credits | $2,883 | |||||||||
| Federal refundable credits | $0 | |||||||||
| Free school meals eligibility | No | |||||||||
| Head CHIP eligibility | No | |||||||||
| Head Medicaid eligibility | No | |||||||||
| Head Medicare eligibility | Yes | |||||||||
| Head WIC eligibility | No | |||||||||
| Local income tax | $0 | |||||||||
| Payroll tax | $0 | |||||||||
| Reduced-price school meals eligibility | No | |||||||||
| Self-employment tax | $0 | |||||||||
| SNAP | $0 | |||||||||
| SSI | $0 | |||||||||
| State tax before refundable credits | $0 | |||||||||
| State refundable credits | $0 | |||||||||
| TANF | $0 |
Loading explanation and audit text…
Exact promptFull household batch contract for all benchmark outputsProvider-specific structured-output transport, no external tool
Estimate the requested tax and benefit outputs using only the household facts below. All listed people live together and are in one household group for tax and benefit calculations. All listed facts describe the full tax-benefit year. Treat demographic, work, student, disability, housing, health coverage, and household-composition facts as constant throughout the tax-benefit year, with no within-year income volatility or status changes. Gross wage and salary amounts are annual totals, including any overtime pay; hourly wage is a straight-time rate when listed. Treat any unlisted numeric input as 0 and any other unlisted household fact, boolean, or status input as false. Assume tax filing and program take-up when required. Do not infer unlisted income, expenses, assets, benefit receipt, rent, or health coverage. Household: - state: TX - tax year: 2026 Head: - age: 77 - bank account assets: $2,580 - employer sponsored insurance premiums: $8,389 - has employer-sponsored insurance - is a surviving spouse - real estate taxes: $16,470 - Social Security retirement income: $26,096 - tax-exempt interest income: $5,578 - taxable interest income: $45 - taxable IRA distributions: $7,849 - taxable private pension income: $24,000 - veterans benefits: $48,000 Tax unit: - first home mortgage balance: $1,000 Provide the following policy quantities for this household: - federal_income_tax_before_refundable_credits: federal individual income tax after nonrefundable credits and before refundable credits. This subtracts nonrefundable credits actually used, including CDCC and the nonrefundable portion of CTC or other credits when applicable; it does not subtract EITC or refundable portions of credits such as refundable CTC - federal_refundable_credits: total refundable federal income tax credits, including EITC and refundable portions of credits such as refundable CTC when applicable; exclude the ACA Premium Tax Credit - payroll_tax: annual household employee-side payroll tax: employee Social Security tax, employee Medicare tax, Additional Medicare Tax, and mandatory employee state payroll taxes. Exclude employer payroll taxes, FUTA, employer unemployment-insurance taxes, and self-employment tax - self_employment_tax: annual self-employment tax liability, excluding employee payroll taxes and Additional Medicare Tax - state_income_tax_before_refundable_credits: state individual income tax after nonrefundable credits and before refundable credits, excluding local income and payroll taxes - state_refundable_credits: total refundable state individual income tax credits - local_income_tax: annual local income, wage, and earnings tax liability in the separate local-income-tax output: NYC income tax, Philadelphia wage tax, Kansas City earnings tax, and St. Louis earnings tax where applicable - snap: annual SNAP (food stamps) benefit amount - ssi: annual Supplemental Security Income (SSI) amount - tanf: annual Temporary Assistance for Needy Families (TANF) benefit amount - head_wic_eligible: whether Head is eligible for WIC (1 if yes, 0 if no) - head_medicaid_eligible: whether Head is eligible for Medicaid under PolicyEngine rules, not whether they are currently enrolled (1 if yes, 0 if no) - head_chip_eligible: whether Head is eligible for CHIP under PolicyEngine rules, not whether they are currently enrolled (1 if yes, 0 if no) - head_medicare_eligible: whether Head is eligible for Medicare (1 if yes, 0 if no) - free_school_meals_eligible: whether PolicyEngine returns positive annual free school meal support for the household (1 if yes, 0 if no; reduced-price meals do not count as 1) - reduced_price_school_meals_eligible: whether PolicyEngine returns positive annual reduced-price school meal support for the household (1 if yes, 0 if no; free meals do not count as 1) Use the `submit_outputs` function exactly once. Return an `outputs` object with every requested quantity keyed by variable name. Each requested key must map to an object with a numeric `value` and a non-empty, specific, concise `explanation`. Each explanation must support the numeric value submitted for the same variable in `outputs`. If an explanation mentions a final amount, that amount must match the corresponding `outputs` value. Do not write that you will use one value while submitting a different value. Do not include scratch work, abandoned calculations, or corrections. End each explanation with `value = X`, where X exactly matches the numeric `value` field. For 1/0 eligibility outputs, submit 1 only when the explanation says eligible or yes, and submit 0 only when it says not eligible or no. Use the exact variable names as keys inside `outputs`. Include every requested key exactly once in `outputs`, even if the value is 0. Put only numeric values in `value`, with no dollar signs, commas, or explanatory text. Do not rely on plain text for the final answers. If an answer is a currency amount, give the annual amount. If an answer is a rate, give a decimal (e.g. 0.25 for 25%).
Where models still break
The hardest part of PolicyBench is not saying when a program is zero. It is getting the positive amount right for the households that actually qualify. The cards below split those cases apart so the benchmark is not flattered by easy zero-answer rows.
These cards are intentionally stricter than the aggregate leaderboard but still use within-10% accuracy for dollar-valued programs so positive cases stay interpretable. Positive-amount cases is the harder and more informative number for benefits and refundable credits. For binary coverage flags, the cards compare positive and negative class accuracy.
These expanders summarize recurring miss patterns from direct reads of model answers and explanations. They sit here with failure modes because they describe why the low-scoring program slices break.
Federal tax before refundable creditsThis target isolates federal income tax after nonrefundable credits but before refundable credits.
Avg 75%
This target isolates federal income tax after nonrefundable credits but before refundable credits.
- It subtracts nonrefundable credits actually used, such as CDCC and the nonrefundable part of CTC when applicable.
- It leaves EITC and refundable credit portions, such as refundable CTC, for the refundable-credits output.
State tax before refundable creditsThis output usually fails when models import rough federal or flat-rate logic into state-specific tax bases.
Avg 77%
This output usually fails when models import rough federal or flat-rate logic into state-specific tax bases.
- In small-liability cases like scenarios_055 and _060, several models overshoot by a wide margin relative to the reference value.
- In large cases like scenario_042, the main failure is still the wrong state tax base rather than the final credit step.
State refundable creditsMost rows are easy zeros; the informative misses are the few positive state credits that models leave at zero.
Avg 81%
Most rows are easy zeros; the informative misses are the few positive state credits that models leave at zero.
- In Colorado scenario_090, several models return $0 against a $6,836 reference value.
- When models do predict a positive state credit, they often derive it from a rough federal-credit ratio instead of the state program itself.
SNAPPositive SNAP cases are the main miss; many models zero them out using raw asset or net-worth heuristics.
Avg 84%
Positive SNAP cases are the main miss; many models zero them out using raw asset or net-worth heuristics.
- In scenarios_035 and _047, several models return $0 on households with reference values above $11,000.
- In scenario_092, Gemini Pro cites SNAP asset limits and returns $0 even though the visible prompt inputs do not support that denial.
Federal refundable creditsThis target captures the refundable federal credit side of the income-tax calculation.
Avg 88%
This target captures the refundable federal credit side of the income-tax calculation.
- It includes EITC and refundable portions of credits such as refundable CTC when applicable.
- It keeps refundable income-tax credits separate from the nonrefundable-credit target.
Payroll taxThis target combines multiple policy rules, and errors usually come from positive cases rather than zero cases.
Avg 92%
This target combines multiple policy rules, and errors usually come from positive cases rather than zero cases.
Person-level Medicaid eligibilityModels often overuse Medicare enrollment or visible assets as disqualifiers and miss non-wage eligibility pathways.
Avg 88%
Models often overuse Medicare enrollment or visible assets as disqualifiers and miss non-wage eligibility pathways.
- In scenarios_054 and _076, many models return 0 when the reference flag is 1.
- The errors are not just arithmetic. They reflect the wrong eligibility pathway being chosen from the household facts.
Person-level Medicare eligibilityModels often overuse Medicare enrollment or visible assets as disqualifiers and miss non-wage eligibility pathways.
Avg 95%
Models often overuse Medicare enrollment or visible assets as disqualifiers and miss non-wage eligibility pathways.
- In scenarios_054 and _076, many models return 0 when the reference flag is 1.
- The errors are not just arithmetic. They reflect the wrong eligibility pathway being chosen from the household facts.
Person-level CHIP eligibilityModels often overuse Medicare enrollment or visible assets as disqualifiers and miss non-wage eligibility pathways.
Avg 95%
Models often overuse Medicare enrollment or visible assets as disqualifiers and miss non-wage eligibility pathways.
- In scenarios_054 and _076, many models return 0 when the reference flag is 1.
- The errors are not just arithmetic. They reflect the wrong eligibility pathway being chosen from the household facts.
Person-level Head Start eligibilityThis target combines multiple policy rules, and errors usually come from positive cases rather than zero cases.
Avg 96%
This target combines multiple policy rules, and errors usually come from positive cases rather than zero cases.
How the United States benchmark works
PolicyBench measures a no-tools task: how well frontier models can estimate person- and household-level tax and benefit outputs from the prompt alone while following a structured response contract. This app shows the current no-tools US benchmark on a fixed test set, with PolicyEngine reference outputs computed by PolicyEngine-US for tax year 2026.
max(0, 1 − |pred − ref| / |ref|)when the reference is nonzero and exact zero matches when the reference is zero for amount outputs, and the same exact 0/1 rule for binary outputs. Each full source household's per-output share is |ref| / max(|household_net_income|, Σ |ref|), a value in [0, 1] that's strictly less than one when net income dominates the gross tax-benefit flow and equals one only when programs cancel each other out. Those shares are averaged using calibrated household weights in the full weighting population, then renormalized so the output weights sum to one. US weights use the full populace dataset. The weights are then applied to the fixed benchmark households and renormalized within each household over requested outputs. Person-level eligibility flags like Medicaid carry weight through PolicyEngine's paired per-capita value (e.g. medicaid_value), so the LLM is graded only on the boolean call itself. Missing or unparseable answers count as misses through the coverage multiplier. The leaderboard reports the exact-match rate as the headline deployability bar, with within-1% as a near-miss-tolerant companion and bounded score, amount accuracy, and participation accuracy as further diagnostics. Equal-weight and budget-weighted variants are reported alongside for transparency. The leaderboard is a point estimate on this fixed test set.