PolicyBench stops scoring eight tax outputs whose references turn on facts the prompts never state

PolicyBench stops scoring eight tax outputs, in six households, whose references turn on facts the prompts never state. The eight are federal income tax for households in California, Massachusetts, and Virginia; the Virginia household's state income tax; and payroll tax for the Massachusetts household and households in Minnesota, Colorado, and New York. Every model is now scored on 1,920 of its 1,984 requested outputs, down from 1,928, and PolicyBench excludes 64.

The three federal outputs depend on the state income tax each household paid during 2026, which the federal deduction for state and local taxes counts and no prompt lists. policyengine-us 2.15.17, which built the references, fills that amount with its own estimate of state income tax withheld, a formula on federal adjusted gross income. If each household paid its 2026 state income tax as the engine computes it, the outputs are $11,895.23 in California, $26,991.31 in Massachusetts, and $11,137.31 in Virginia, against references of $11,113.57, $26,932.27, and $10,729.61.

The Virginia household's head is 69, and the prompt lists no Medicare enrollment or premium. policyengine-us treats everyone eligible for Medicare as enrolled and counts the standard Part B premium, $2,434.80 for 2026, as a medical expense. The household itemizes, and Virginia's itemized deductions start from the federal ones, so the premium lowers its Virginia income tax: without it, that output is $3,654.15 rather than the reference's $3,514.15. The premium lowers the household's federal income tax too, and that output's exclusion record names both unstated facts.

The four payroll outputs count an employee share of a state paid-leave or disability premium: Minnesota Paid Leave, Colorado FAMLI, Massachusetts Paid Family and Medical Leave, and New York Paid Family Leave and Disability Benefits. In each of these states the premium is the employer's, and the employer may deduct up to a set share of it from wages but need not, so the law requires no amount of the employee. The prompts ask for mandatory employee state payroll taxes, and the references count the largest share each state's published 2026 rates let the employer deduct, without the prompts saying whether it does. Without that share, the outputs are $2,218.50 in Minnesota, $312.74 in Colorado, $13,387.65 in Massachusetts, and $7,664.92 in New York, against references of $2,346.10, $330.73, $14,192.66, and $8,108.03. PolicyBench still scores the payroll outputs of seven households in California, Connecticut, and Pennsylvania, where the law puts the contribution on the worker and requires the employer to withhold it.

No model's answer is within $1 of the reference on any of the four income tax outputs. On the four payroll outputs, 35 answers match the reference within $1: six for Minnesota, 16 for Colorado, nine for Massachusetts, and four for New York. Another 82 match the amount without the state share.

Every model's exact rate, its share of answers within $1 weighted by household impact, rises by 0.67 to 1.35 points. GPT-6 Sol still leads at 95.8%, ahead of Claude Opus 5.5 (95.1%) and GPT-5.6 Sol (94.4%). Claude Sonnet 5.5 (93.3%) passes GPT-6 Luna (93.0%) for fourth. Further down, Inkling moves above Grok 4.7, Gemini 3 Flash Preview above Claude Opus 4.7, and Gemini 3.1 Flash Lite Preview above DeepSeek V4 Pro. Every other pair keeps its order.

PolicyBench's next prompt contract remains an unactivated draft (PolicyEngine/policybench#173). It states the state income tax paid for each tax unit or a named convention for estimating withholding; each person's Medicare Part B enrollment and annual net premium; and whether each person's employer withholds the employee share of state paid-leave or disability premiums. Its payroll output definition counts optional employee shares only when the employer withholds them. A later run can score these outputs after its reference builder and a fresh input sweep validate the stated conventions.

Two corrections in this release move no score. PolicyBench now reports that it collected model responses from June 12 to September 29, 2026, ending on the UTC date of the last answer; release dashboard-data-20260930 gave September 30, the date of that release. The audit text for a California household head's Medicaid eligibility now explains the head's income test as the engine applies it: California's $230 monthly income disregard, which takes the place of SSI's $20 general income exclusion, brings countable income under the limit. The earlier text said the engine subtracts a Medicare Part B premium from income, which it does not.