Skip to main content
PolicyBench

Testing how accurately language models calculate household taxes and benefits.

23Models
100Households
18Outputs
Snapshot 2026-07-09

PolicyBench leaderboard

Leaderboard

Model rankings

23 models, ranked by exact-match score (United States)

$0.090 per household
88.7%
$0.019 per household
84.5%
3GPT-5.5
$0.262 per household
83.5%
$0.039 per household
83.4%
5Claude Fable 5
$0.541 per household
79.9%
$0.085 per household
77.9%
7Claude Opus 4.7
$0.293 per household
77.4%
$0.015 per household
77.2%
$0.186 per household
77.1%
$0.048 per household
76.9%
$0.056 per household
76.2%
$0.045 per household
76.1%
$0.002 per household
76.1%
14DeepSeek V4 Pro
$0.011 per household
76.1%
15Qwen 3.7 Max
$0.139 per household
73.6%
16GLM-5.2
$0.082 per household
73.1%
17Claude Opus 4.8
$0.255 per household
72.6%
18MiniMax M3
$0.007 per household
72.4%
19Claude Haiku 4.5
$0.055 per household
71.7%
20GPT-5.4 mini
$0.006 per household
70.5%
21Claude Sonnet 5
$0.172 per household
69.4%
22Kimi K2.6
$0.391 per household
64.6%
23GPT-5.4 nano
$0.002 per household
62.3%
Optionsscoring: exact · cases: all cases · weighting: household · all 18 programs
Programs

Filter restricts model scoring and the program breakdown table to the selected outputs. The scenario explorer remains unfiltered so each household's full prompt stays visible. Weights rescale to 100% over the active set.

Scoring
Percent matching within $1.
Reference cases
All reference cells.
Weighting
Population household-impact weights — each output group's share is |ref| / max(|household_net_income|, Σ|ref|) in the full weighting population, averaged with household weights and renormalized before scoring each benchmark household. US weights use the full populace dataset.
Per-variable weights18 variables, sorted by Household
VariableHouseholdAggregateEqual
Person-level Medicaid eligibility29.86%16.02%5.56%
Federal tax before refundable credits19.14%42.18%5.56%
Payroll tax15.65%14.34%5.56%
Person-level Medicare eligibility10.74%5.42%5.56%
State tax before refundable credits6.00%10.37%5.56%
SNAP4.15%1.74%5.56%
Federal refundable credits3.70%2.13%5.56%
Person-level Early Head Start eligibility3.10%2.39%5.56%
Self-employment tax2.09%1.56%5.56%
SSI1.97%1.08%5.56%
Person-level Head Start eligibility1.18%0.84%5.56%
Free school meals eligibility0.74%0.57%5.56%
State refundable credits0.55%0.35%5.56%
Person-level WIC eligibility0.32%0.23%5.56%
Local income tax0.26%0.43%5.56%
TANF0.26%0.16%5.56%
Person-level CHIP eligibility0.18%0.12%5.56%
Reduced-price school meals eligibility0.10%0.05%5.56%
By program

Program breakdown

Bounded score by program and model (AI alone). Dollar targets use continuous relative-error partial credit; binary coverage flags use exact accuracy.

Program filterAll 18 programs

Shared with model scoring. The scenario explorer remains unfiltered so each household's full prompt stays visible. The table shows only selected outputs; model scores rescale selected weights to 100%.

ProgramClaude Fable 5Claude Opus 4.8Claude Opus 4.7Claude Sonnet 5Claude Sonnet 4.6Claude Haiku 4.5Grok 4.3Grok Build 0.1GPT-5.6 SolGPT-5.6 TerraGPT-5.6 LunaGPT-5.5GPT-5.4 miniGPT-5.4 nanoGemini 3.1 Pro PreviewGemini 3.5 FlashGemini 3 Flash PreviewGemini 3.1 Flash Lite PreviewDeepSeek V4 ProKimi K2.6GLM-5.2MiniMax M3Qwen 3.7 MaxAvg
Local income tax
100%
100%
100%
99%
100%
99%
100%
100%
100%
100%
100%
100%
100%
100%
100%
99%
98%
99%
100%
87%
94%
100%
100%
99%
TANF
99%
100%
99%
100%
99%
99%
99%
100%
100%
99%
100%
100%
99%
99%
99%
100%
100%
99%
100%
89%
94%
99%
99%
99%
Free school meals eligibility
99%
97%
99%
99%
98%
98%
99%
98%
100%
98%
98%
98%
98%
92%
99%
100%
100%
99%
98%
91%
93%
99%
99%
98%
Reduced-price school meals eligibility
100%
94%
98%
99%
98%
92%
99%
97%
99%
99%
97%
99%
99%
99%
99%
98%
98%
99%
99%
90%
94%
99%
98%
98%
Person-level Early Head Start eligibility
100%
97%
100%
100%
95%
87%
97%
100%
100%
100%
100%
100%
92%
97%
100%
100%
100%
97%
100%
95%
89%
97%
97%
97%
Person-level WIC eligibility
99%
92%
95%
94%
99%
93%
98%
100%
100%
99%
100%
100%
86%
96%
99%
100%
99%
99%
99%
91%
93%
97%
99%
97%
SSI
98%
98%
98%
98%
99%
97%
97%
97%
97%
98%
98%
96%
97%
97%
97%
98%
98%
97%
98%
87%
91%
96%
98%
97%
Self-employment tax
99%
100%
100%
97%
99%
93%
99%
99%
100%
98%
99%
100%
96%
96%
100%
100%
99%
98%
99%
60%
90%
96%
96%
96%
Person-level Head Start eligibility
100%
87%
87%
95%
97%
97%
97%
100%
100%
100%
100%
100%
84%
95%
95%
100%
100%
97%
97%
97%
89%
95%
89%
96%
Person-level Medicare eligibility
89%
94%
90%
96%
94%
97%
98%
98%
94%
99%
97%
90%
96%
91%
97%
97%
97%
97%
98%
90%
89%
97%
97%
95%
Person-level CHIP eligibility
99%
88%
88%
92%
89%
90%
96%
98%
99%
100%
97%
95%
92%
100%
98%
96%
99%
100%
98%
84%
93%
98%
90%
95%
Payroll tax
98%
98%
99%
98%
100%
91%
97%
97%
99%
99%
98%
99%
82%
63%
94%
93%
95%
98%
95%
48%
88%
84%
97%
92%
Person-level Medicaid eligibility
94%
78%
86%
75%
87%
82%
89%
94%
97%
91%
94%
95%
84%
71%
97%
97%
95%
90%
94%
82%
86%
84%
88%
88%
Federal refundable credits
93%
88%
89%
84%
95%
80%
86%
90%
99%
96%
96%
96%
84%
86%
93%
92%
93%
89%
92%
46%
83%
82%
82%
88%
SNAP
86%
78%
81%
75%
84%
81%
79%
86%
89%
88%
87%
87%
77%
80%
87%
90%
88%
84%
87%
79%
84%
82%
83%
84%
State refundable credits
86%
81%
81%
80%
82%
79%
79%
80%
87%
82%
84%
89%
79%
79%
86%
86%
89%
80%
84%
48%
74%
80%
80%
81%
State tax before refundable credits
80%
84%
85%
80%
84%
71%
81%
75%
89%
84%
79%
90%
69%
65%
86%
82%
84%
72%
79%
51%
66%
63%
71%
77%
Federal tax before refundable credits
86%
78%
86%
72%
86%
71%
75%
71%
93%
91%
87%
88%
63%
61%
77%
75%
76%
68%
74%
48%
77%
58%
66%
75%
Cells use color as a redundant cue; the percentage shown in each cell is the actual benchmark score.
<50%
50-59%
60-69%
70-79%
80-89%
90%+
Deep dive

Scenario explorer

Inspect benchmark households, reference outputs, model answers, and the exact prompt sent to every model.

Show
Provider
ProgramReference
Fable 5
Grok 4.3
GPT-5.6 Sol
Pro Preview
V4 Pro
Kimi K2.6
GLM-5.2
MiniMax M3
3.7 Max
Federal tax before refundable credits$2,883
Federal refundable credits$0
Free school meals eligibilityNo
Head CHIP eligibilityNo
Head Medicaid eligibilityNo
Head Medicare eligibilityYes
Head WIC eligibilityNo
Local income tax$0
Payroll tax$0
Reduced-price school meals eligibilityNo
Self-employment tax$0
SNAP$0
SSI$0
State tax before refundable credits$0
State refundable credits$0
TANF$0
Explanation and audit coverage

Loading explanation and audit text…

LLM: 23parse contract failure: 3
Exact prompt
Full household batch contract for all benchmark outputs
Provider-specific structured-output transport, no external tool
Estimate the requested tax and benefit outputs using only the household facts below. All listed people live together and are in one household group for tax and benefit calculations. All listed facts describe the full tax-benefit year. Treat demographic, work, student, disability, housing, health coverage, and household-composition facts as constant throughout the tax-benefit year, with no within-year income volatility or status changes. Gross wage and salary amounts are annual totals, including any overtime pay; hourly wage is a straight-time rate when listed. Treat any unlisted numeric input as 0 and any other unlisted household fact, boolean, or status input as false. Assume tax filing and program take-up when required. Do not infer unlisted income, expenses, assets, benefit receipt, rent, or health coverage.

Household:
- state: TX
- tax year: 2026

Head:
- age: 77
- bank account assets: $2,580
- employer sponsored insurance premiums: $8,389
- has employer-sponsored insurance
- is a surviving spouse
- real estate taxes: $16,470
- Social Security retirement income: $26,096
- tax-exempt interest income: $5,578
- taxable interest income: $45
- taxable IRA distributions: $7,849
- taxable private pension income: $24,000
- veterans benefits: $48,000

Tax unit:
- first home mortgage balance: $1,000

Provide the following policy quantities for this household:
- federal_income_tax_before_refundable_credits: federal individual income tax after nonrefundable credits and before refundable credits. This subtracts nonrefundable credits actually used, including CDCC and the nonrefundable portion of CTC or other credits when applicable; it does not subtract EITC or refundable portions of credits such as refundable CTC
- federal_refundable_credits: total refundable federal income tax credits, including EITC and refundable portions of credits such as refundable CTC when applicable; exclude the ACA Premium Tax Credit
- payroll_tax: annual household employee-side payroll tax: employee Social Security tax, employee Medicare tax, Additional Medicare Tax, and mandatory employee state payroll taxes. Exclude employer payroll taxes, FUTA, employer unemployment-insurance taxes, and self-employment tax
- self_employment_tax: annual self-employment tax liability, excluding employee payroll taxes and Additional Medicare Tax
- state_income_tax_before_refundable_credits: state individual income tax after nonrefundable credits and before refundable credits, excluding local income and payroll taxes
- state_refundable_credits: total refundable state individual income tax credits
- local_income_tax: annual local income, wage, and earnings tax liability in the separate local-income-tax output: NYC income tax, Philadelphia wage tax, Kansas City earnings tax, and St. Louis earnings tax where applicable
- snap: annual SNAP (food stamps) benefit amount
- ssi: annual Supplemental Security Income (SSI) amount
- tanf: annual Temporary Assistance for Needy Families (TANF) benefit amount
- head_wic_eligible: whether Head is eligible for WIC (1 if yes, 0 if no)
- head_medicaid_eligible: whether Head is eligible for Medicaid under PolicyEngine rules, not whether they are currently enrolled (1 if yes, 0 if no)
- head_chip_eligible: whether Head is eligible for CHIP under PolicyEngine rules, not whether they are currently enrolled (1 if yes, 0 if no)
- head_medicare_eligible: whether Head is eligible for Medicare (1 if yes, 0 if no)
- free_school_meals_eligible: whether PolicyEngine returns positive annual free school meal support for the household (1 if yes, 0 if no; reduced-price meals do not count as 1)
- reduced_price_school_meals_eligible: whether PolicyEngine returns positive annual reduced-price school meal support for the household (1 if yes, 0 if no; free meals do not count as 1)

Use the `submit_outputs` function exactly once. Return an `outputs` object with every requested quantity keyed by variable name. Each requested key must map to an object with a numeric `value` and a non-empty, specific, concise `explanation`. Each explanation must support the numeric value submitted for the same variable in `outputs`. If an explanation mentions a final amount, that amount must match the corresponding `outputs` value. Do not write that you will use one value while submitting a different value. Do not include scratch work, abandoned calculations, or corrections. End each explanation with `value = X`, where X exactly matches the numeric `value` field. For 1/0 eligibility outputs, submit 1 only when the explanation says eligible or yes, and submit 0 only when it says not eligible or no. Use the exact variable names as keys inside `outputs`. Include every requested key exactly once in `outputs`, even if the value is 0. Put only numeric values in `value`, with no dollar signs, commas, or explanatory text. Do not rely on plain text for the final answers. If an answer is a currency amount, give the annual amount. If an answer is a rate, give a decimal (e.g. 0.25 for 25%).
Household #000
Full household facts

Exactly the household section the models see at the top of the prompt — verbatim, no summarization.

Household:
- state: TX
- tax year: 2026

Head:
- age: 77
- bank account assets: $2,580
- employer sponsored insurance premiums: $8,389
- has employer-sponsored insurance
- is a surviving spouse
- real estate taxes: $16,470
- Social Security retirement income: $26,096
- tax-exempt interest income: $5,578
- taxable interest income: $45
- taxable IRA distributions: $7,849
- taxable private pension income: $24,000
- veterans benefits: $48,000

Tax unit:
- first home mortgage balance: $1,000
Failure modes

Where models still break

The hardest part of PolicyBench is not saying when a program is zero. It is getting the positive amount right for the households that actually qualify. The cards below split those cases apart so the benchmark is not flattered by easy zero-answer rows.

How to read these cards

These cards are intentionally stricter than the aggregate leaderboard but still use within-10% accuracy for dollar-valued programs so positive cases stay interpretable. Positive-amount cases is the harder and more informative number for benefits and refundable credits. For binary coverage flags, the cards compare positive and negative class accuracy.

Dollar target
Federal tax before refundable credits
Overall 63.1%
Positive-amount cases33.9%
Zero-amount cases89.5%
With children54.9%
Low income90.5%
High income36.8%
Underpredict share on positives36.9%
Dollar target
State tax before refundable credits
Overall 67.0%
Positive-amount cases34.5%
Zero-amount cases90.8%
With children55.0%
Low income81.5%
High income59.4%
Underpredict share on positives50.2%
Dollar target
State refundable credits
Overall 80.0%
Positive-amount cases9.0%
Zero-amount cases98.7%
With children65.2%
Low income60.3%
High income96.3%
Underpredict share on positives86.5%
Dollar target
SNAP
Overall 81.3%
Positive-amount cases20.3%
Zero-amount cases96.2%
With children79.2%
Low income43.5%
High income100.0%
Underpredict share on positives70.8%
Dollar target
Federal refundable credits
Overall 87.6%
Positive-amount cases40.1%
Zero-amount cases94.6%
With children59.0%
Low income74.4%
High income98.9%
Underpredict share on positives69.3%
Dollar target
Payroll tax
Overall 88.7%
Positive-amount cases83.8%
Zero-amount cases97.1%
With children83.2%
Low income93.5%
High income77.9%
Underpredict share on positives48.3%
Household boolean
Person-level Medicaid eligibility
Overall 89.1%
Positive households77.3%
Negative households92.8%
With children88.1%
Low income81.9%
High income99.6%
Household boolean
Person-level Medicare eligibility
Overall 95.5%
Positive households99.4%
Negative households94.4%
With children99.3%
Low income94.7%
High income97.9%
Household boolean
Person-level CHIP eligibility
Overall 95.6%
Positive householdsn/a
Negative households95.6%
With children89.2%
Low income92.1%
High income99.9%
Household boolean
Person-level Head Start eligibility
Overall 96.2%
Positive households89.1%
Negative households96.6%
With children96.2%
Low income90.9%
High income98.9%
What the error reads show

These expanders summarize recurring miss patterns from direct reads of model answers and explanations. They sit here with failure modes because they describe why the low-scoring program slices break.

Federal tax before refundable credits

This target isolates federal income tax after nonrefundable credits but before refundable credits.

Avg 75%
Common misses
  • It subtracts nonrefundable credits actually used, such as CDCC and the nonrefundable part of CTC when applicable.
  • It leaves EITC and refundable credit portions, such as refundable CTC, for the refundable-credits output.
State tax before refundable credits

This output usually fails when models import rough federal or flat-rate logic into state-specific tax bases.

Avg 77%
Common misses
  • In small-liability cases like scenarios_055 and _060, several models overshoot by a wide margin relative to the reference value.
  • In large cases like scenario_042, the main failure is still the wrong state tax base rather than the final credit step.
State refundable credits

Most rows are easy zeros; the informative misses are the few positive state credits that models leave at zero.

Avg 81%
Common misses
  • In Colorado scenario_090, several models return $0 against a $6,836 reference value.
  • When models do predict a positive state credit, they often derive it from a rough federal-credit ratio instead of the state program itself.
SNAP

Positive SNAP cases are the main miss; many models zero them out using raw asset or net-worth heuristics.

Avg 84%
Common misses
  • In scenarios_035 and _047, several models return $0 on households with reference values above $11,000.
  • In scenario_092, Gemini Pro cites SNAP asset limits and returns $0 even though the visible prompt inputs do not support that denial.
Federal refundable credits

This target captures the refundable federal credit side of the income-tax calculation.

Avg 88%
Common misses
  • It includes EITC and refundable portions of credits such as refundable CTC when applicable.
  • It keeps refundable income-tax credits separate from the nonrefundable-credit target.
Payroll tax

This target combines multiple policy rules, and errors usually come from positive cases rather than zero cases.

Avg 92%
Common misses
    Person-level Medicaid eligibility

    Models often overuse Medicare enrollment or visible assets as disqualifiers and miss non-wage eligibility pathways.

    Avg 88%
    Common misses
    • In scenarios_054 and _076, many models return 0 when the reference flag is 1.
    • The errors are not just arithmetic. They reflect the wrong eligibility pathway being chosen from the household facts.
    Person-level Medicare eligibility

    Models often overuse Medicare enrollment or visible assets as disqualifiers and miss non-wage eligibility pathways.

    Avg 95%
    Common misses
    • In scenarios_054 and _076, many models return 0 when the reference flag is 1.
    • The errors are not just arithmetic. They reflect the wrong eligibility pathway being chosen from the household facts.
    Person-level CHIP eligibility

    Models often overuse Medicare enrollment or visible assets as disqualifiers and miss non-wage eligibility pathways.

    Avg 95%
    Common misses
    • In scenarios_054 and _076, many models return 0 when the reference flag is 1.
    • The errors are not just arithmetic. They reflect the wrong eligibility pathway being chosen from the household facts.
    Person-level Head Start eligibility

    This target combines multiple policy rules, and errors usually come from positive cases rather than zero cases.

    Avg 96%
    Common misses
      Methodology

      How the United States benchmark works

      PolicyBench measures a no-tools task: how well frontier models can estimate person- and household-level tax and benefit outputs from the prompt alone while following a structured response contract. This app shows the current no-tools US benchmark on a fixed test set, with PolicyEngine reference outputs computed by PolicyEngine-US for tax year 2026.

      100
      populace households
      18
      Scored variables
      1,984
      Model-output targets
      23
      Frontier models
      Task
      Each model sees the same household description and must return all scored outputs plus a short explanation for each output in one response, with no tool use. The exact provider-specific prompts are visible in the scenario explorer, so you can inspect the contract instead of inferring it.
      Open-set status
      The public scenario explorer exposes prompts and PolicyEngine reference outputs, so future model releases or fine-tunes could learn from the released cases. Treat this leaderboard as a public preview; protected held-out claims would require a separate rotating evaluation set.
      Households
      The US benchmark samples households from PolicyEngine's populace US microdataset with a fixed seed. The current set is restricted to households with a single federal tax unit, a single family, and a single benefit-calculation unit. Adult dependents remain in scope when they satisfy those restrictions. Ages, roles, income sources, and other nonzero promptable inputs are carried through into both the prompt and the PolicyEngine-US input; filing status is inferred from household structure.
      Reference outputs
      PolicyEngine-US computes the PolicyEngine reference output for every household-variable pair in tax year 2026. The displayed variables define the benchmark scope for this snapshot.
      Output selection
      The benchmark includes direct tax, credit, benefit, health-support, and coverage outputs that can plausibly be estimated from household facts. It excludes intermediate tax bases, payroll subcomponents, and outputs that mainly require unavailable history, restricted local market data, or program take-up assignment. WIC is scored as person-level eligibility, not as a dollar amount. Local income tax is retained as a displayed requested output, but currently receives zero default population-impact weight because the full populace dataset has no positive modeled local-income-tax records. The source run also requested the ACA Premium Tax Credit, but explanation audits showed the prompt could be misleading when households lacked plan-specific Marketplace information, so it is preserved in raw responses and excluded from the scored leaderboard.
      Scoring and weighting
      The public leaderboard ranks models by the exact-match rate using population household-impact weights. For each household-output row, the within-1% indicator is 1 when a currency answer is within 1% of the PolicyEngine reference value, with a one-currency-unit tolerance when the reference is zero. Binary eligibility flags are requested as integer 0/1 outputs and require exact 0/1 matching. The secondary bounded score uses max(0, 1 − |pred − ref| / |ref|)when the reference is nonzero and exact zero matches when the reference is zero for amount outputs, and the same exact 0/1 rule for binary outputs. Each full source household's per-output share is |ref| / max(|household_net_income|, Σ |ref|), a value in [0, 1] that's strictly less than one when net income dominates the gross tax-benefit flow and equals one only when programs cancel each other out. Those shares are averaged using calibrated household weights in the full weighting population, then renormalized so the output weights sum to one. US weights use the full populace dataset. The weights are then applied to the fixed benchmark households and renormalized within each household over requested outputs. Person-level eligibility flags like Medicaid carry weight through PolicyEngine's paired per-capita value (e.g. medicaid_value), so the LLM is graded only on the boolean call itself. Missing or unparseable answers count as misses through the coverage multiplier. The leaderboard reports the exact-match rate as the headline deployability bar, with within-1% as a near-miss-tolerant companion and bounded score, amount accuracy, and participation accuracy as further diagnostics. Equal-weight and budget-weighted variants are reported alongside for transparency. The leaderboard is a point estimate on this fixed test set.
      Sensitivity checks
      The manuscript reports alternative ranking views for equal-output groups, amount-only outputs, binary coverage, positive-reference cases, zero-reference cases, and country-only results. In the equal-output-group view, person-level outputs are grouped by program before the country average. These checks are used to interpret rank stability; they do not replace the public exact-match leaderboard.
      Impact weighting
      Binary coverage flags have 0/1 labels, but a 0/1 label is not their economic impact. Their leaderboard weights therefore come from PolicyEngine value proxies where available, such as estimated health coverage or nutrition-program value, rather than from the binary label itself. When every reference impact in a household is zero, the household falls back to equal output weights.
      Current benchmark scope
      Latest United States run in this app evaluates GPT-5.6 Sol, GPT-5.6 Luna, GPT-5.5, GPT-5.6 Terra, Claude Fable 5, Grok 4.3, Gemini 3.1 Pro Preview, Claude Opus 4.7, Claude Sonnet 4.6, Gemini 3.1 Flash Lite Preview, Gemini 3 Flash Preview, DeepSeek V4 Pro, Grok Build 0.1, Gemini 3.5 Flash, Qwen 3.7 Max, GLM-5.2, MiniMax M3, Claude Opus 4.8, Claude Haiku 4.5, GPT-5.4 mini, Claude Sonnet 5, Kimi K2.6, GPT-5.4 nano on 1,984 scored outputs.
      Fixed test set, no tools, US tax year 2026
      Federal refundable creditsFederal taxFederal tax before refundable creditsFederal taxFree school meals eligibilityCoverageLocal income taxLocal taxPayroll taxPayroll taxPerson-level CHIP eligibilityCoveragePerson-level Early Head Start eligibilityCoveragePerson-level Head Start eligibilityCoveragePerson-level Medicaid eligibilityCoveragePerson-level Medicare eligibilityCoveragePerson-level WIC eligibilityCoverageReduced-price school meals eligibilityCoverageSelf-employment taxPayroll taxSNAPBenefitsSSIBenefitsState refundable creditsState taxState tax before refundable creditsState taxTANFBenefits