Notes

PolicyBench gave three models a tool that runs PolicyEngine on a household the model writes from the prompt's facts, and ran its 100 US households again on October 14 and 15. GPT-6 Sol, which leads the board at 95.2% of answers within $1, weighted by household impact, gets 99.2% with the tool. Claude Opus 5.5 goes from 94.1% to 99.8%, and Gemini 4 Argon from 88.7% to 99.9%. The runs are a pilot, not board rows, since the board scores models answering alone, and the result is circular: the tool runs the engine that computes PolicyBench's references, so the scores measure agreement with that engine, not whether it is right.

Most of the answers the tool fixes are dollar amounts that are not zero. Of the 187 nonzero amounts, GPT-6 Sol gets 67.4% within $1 on its own and 96.8% with the tool; Claude Opus 5.5, 70.1% and 98.9%; and Gemini 4 Argon, 46.5% and 99.5%.

With the tool, 12 answers are wrong across the three runs: nine of GPT-6 Sol's, two of Claude Opus 5.5's and one of Gemini 4 Argon's. Each traces to the household the model described to the engine, or to the output it read back, rather than to the calculation: in seven the model had entered a stated fact under another variable, in three it had left one out, and in two it read back the wrong output. Prompt labels share the blame for two: "employer sponsored insurance premiums" never says the employer paid them, and "disability benefits" never says they are not Social Security. In five of GPT-6 Sol's, it had used all eight calculation turns the pilot allows. GPT-6 Sol is also the only model that turned right answers wrong, two of them.

The tool costs more. For the 100 households, GPT-6 Sol costs $5.63 with it and $2.71 without, Claude Opus 5.5 $7.99 and $6.75, and Gemini 4 Argon $8.01 and $2.79.

The tool applies the conventions PolicyBench uses for its references, so a model that describes a household exactly as the reference does gets the reference's answer. Describing a household is also easier here than it would be from a filer's documents: 1,065 of the 1,632 facts the prompts state, not counting ages and wages, carry a label that is the engine's own variable name with spaces.

Each model's answers with the tool, its calculations and cost per household, the remaining misses with the inputs behind them, and the script that scores them the board's way are in the data below.

America.gov, the General Services Administration's assistant, live since September 29, gets 87.9% of PolicyBench answers within $1, weighted by household impact. On the board it would rank #14 of 51, level with GPT-5.5 (87.9%); GPT-6 Sol leads at 95.2%. It does not join the board, which scores models answering alone, with no tools and no web access. America.gov says it uses "Super Intelligence (SI) to find and explain information from official government sources."

Between October 14 and 18, PolicyBench sent each of its 100 households to America.gov in a new chat, with the board's own prompt in the JSON answer format, and scored every reply as it scores the board. Six prompts run past the 8,000 characters America.gov asks a message to stay within, so each went in as two chats asking for half the outputs apiece. Before freezing a run, PolicyBench asks a board model again when its response leaves answers out, and America.gov got the same kind of retry: each refusal went back word for word in a fresh chat, up to twice. On first attempts alone it gets 86.6% (#19).

America.gov's first replies included five refusals. "I cannot estimate or calculate federal or state tax, credit, payroll, or benefit amounts," it replied to the prompt for a Virginia single parent of two. Asked again, two of the refusals turned into full answers and three came back word for word the same, which leaves 3.7% of its scored answers blank; 46 of the 50 models on the board answer every one.

America.gov is strongest on taxes. It gets 72.6% of federal income tax amounts within $1, against a median of 54.8% across the board's models, and 66.7% of state income tax amounts, against 60.2%. Benefits are weaker. On SNAP it gets 81.7%, below the median of 84.9%, and it put a Michigan couple's SNAP at $6,432 a year where the reference and 49 of the 50 models say $0. On Medicaid eligibility, the output that carries the most weight, it gets 92.5% against a median of 95.4%, and it called both adults in a New York couple eligible where the reference and 49 models say neither is.

The current prompts define four outputs by PolicyEngine: Medicaid and CHIP eligibility "under PolicyEngine rules," and the two school-meal outputs as whether "PolicyEngine returns" support. PolicyBench is revising that wording. At least two of America.gov's five refusals cited PolicyEngine; no model on the board declined a household on that ground, though some models' recorded reasoning notes that they cannot run PolicyEngine. Counting only the answers each gave, America.gov is never more than 0.8 points below the median model on those four outputs, and without them it would rank #13 of 51.

This is one run, from one week in October, and America.gov's terms say answers generated with SI "can be incomplete or wrong." The board's last repair step, asking again for each answer still missing, one at a time, would take at least 73 more chats and could lift America.gov to 89.8% at most. The replies, the scores and the method, including how the long prompts were split, are in the data below.

Google's Gemini 4 Argon joined the board on 2026-10-14, the day after Google opened it to the public Gemini API, and Meta's Muse Spark 1.3 joined with it, bringing the board to 50 models. Gemini 4 Argon gets 88.7% of answers within $1, weighted by household impact, #12 of 50 and the highest of any Gemini model; Gemini 3.8 Flash, the previous best, gets 85.6% (#21). Muse Spark 1.3 gets 87.2% (#18), behind Meta's Muse Wonder (90.1%, #10), but on the bounded score, which gives partial credit for amounts close to the reference, it ranks #6. GPT-6 Sol still leads on answers within $1 at 95.2%, ahead of Claude Opus 5.5 (94.1%) and GPT-5.6 Sol (93.1%).

Google announced Gemini 4 Argon on September 30 for a group of partners and released it on the Gemini API on October 13. PolicyBench runs it there, as it does every Gemini model. The API accepts a forced answer tool, so the row answers through the board's tool and reasons at Google's default setting. A forced tool call suppresses some models' reasoning, as it does Claude Opus 5's, so PolicyBench fixed a test before looking at the results and ran all 100 households both ways. With the forced tool Gemini 4 Argon returned reasoning tokens on 80 of them, and on 85 in the board's own run; with an automatic tool choice, on 82. The test found no significant reduction, so the board keeps the forced tool and publishes no second row. At Google's introductory price of $2 per million input tokens and $10 per million output tokens, which doubles on December 1, the 100 households cost $2.79.

Muse Spark 1.3, which Meta released on September 2, is Meta's second model on the board. PolicyBench reaches it through OpenRouter, whose one endpoint for it is Meta's own API, as it does Muse Wonder. That API rejects a forced tool call, so the row answers as a JSON object and reasons at Meta's default setting.

Nothing else on the board moves. The newest policyengine-us release, 2.42.4, gives the same value as 2.41.0 for every output, so the references stay where they were, and every model is still scored on 1,935 of its 1,984 requested outputs. Each of the 48 earlier models keeps its score.

Meta's Muse Wonder joined the board on 2026-10-13, the day after Meta released it, bringing the board to 48 models. It is the first Meta model on the board. It gets 90.1% of answers within $1, weighted by household impact, #10 of 48. On the bounded score, which gives partial credit for amounts close to the reference, it ranks #3, behind only GPT-6 Sol and Claude Opus 5.5. GPT-6 Sol still leads on answers within $1 at 95.2%, ahead of Claude Opus 5.5 (94.1%) and GPT-5.6 Sol (93.1%).

Meta released Muse Wonder Full the same day, but only to Muse Code subscribers, with no API. PolicyBench runs every model through its provider's API under one protocol, so Muse Wonder Full is not on the board. PolicyBench reaches Muse Wonder through OpenRouter, whose one endpoint for it is Meta's own API. That API accepts only an automatic tool choice and rejects a forced tool call, so the row answers as a JSON object, as Claude Opus 5.5's and Claude Sonnet 5.5's do, and reasons at Meta's default setting.

The release also moves PolicyBench's references from policyengine-us 2.38.6 to 2.41.0, the first release with every fix the previous release was waiting on. Of the outputs the previous release excluded, nine return to scoring at the new version's values, each within $1 of the corrected value PolicyBench's audits computed for it. Between them they take the upstream fixes for Idaho's subtraction for health insurance premiums (policyengine-us #10023; two outputs), Wisconsin's retirement income exclusion before refundable credits and Wisconsin's capital gain subtraction for distributions (policyengine-us #10026; two outputs), the IRA deduction's active-participant phase-out, California's itemized deduction conformity and California's educator expense addback (policyengine-us #10032, #10034, #10052) and elective deferrals in the earned income behind the EITC and the refundable child tax credit (policyengine-us #10030; four outputs). The new version also fixes the defect behind a SNAP output, the heat-and-eat utility allowance, but that output stays excluded because the stated facts do not decide it. No reference PolicyBench already scored changes, and nothing is newly excluded. Every model is now scored on 1,935 of its 1,984 requested outputs; PolicyBench excludes the other 49.

Against the previous release, the exact rate of every one of the 47 earlier models falls, by 0.18 to 0.42 points. The cause is the nine restored outputs, which few models get right: half of them are answered within $1 by at most one of the 48 models, and four by none. Muse Wonder answers one of them within $1. Without them, every earlier model's rate would be the previous release's.

Claude Haiku 5.5 joined the board on 2026-10-10, bringing it to 47 models. It gets 79.6% of answers within $1, weighted by household impact, #37 of 47 and 4.8 points above Claude Haiku 4.5 (74.8%, #43). GPT-6 Sol still leads at 95.6%, ahead of Claude Opus 5.5 (94.4%) and GPT-5.6 Sol (93.5%).

Claude Haiku 5.5 cost $0.0013 a household, the least of any row with a cost above zero; Claude Haiku 4.5 cost $0.0554. Unlike Claude Opus 5.5 and Claude Sonnet 5.5, its API accepts a forced tool call, so its row takes the board's forced answer tool. A probe with that request returned no thinking block, where a request without the forced tool did, so the row answers without extended thinking, as Claude Opus 5's does. Re-run with tool_choice: auto, it gets 89.2%, which would rank #10.

The release also moves PolicyBench's references from policyengine-us 2.15.17 to 2.38.6, a release that fixes the engine defects behind 18 outputs. The previous release excluded 14 of them, and they return to scoring at the new version's values. The other four are the October 6 defects below, which stay scored at the new version's values. Each of the 18 lands within $1 of the corrected value PolicyBench's audits computed for it. Between them they take the upstream fixes for the IRA deduction's compensation limit and a dependent's contributions (policyengine-us #10032, #9801; three outputs), the IRA deduction's active-participant phase-out and California's itemized deduction conformity (policyengine-us #10032, #10034), New Jersey's worker unemployment and workforce contributions (policyengine-us #10031), Arizona's standard deduction indexing (policyengine-us #9928), California's itemized deduction conformity (policyengine-us #10034), Ohio's medical deduction for health insurance premiums (policyengine-us #10020), estate income in gross income (policyengine-us #10027, #9633; two outputs), Colorado's 2026 sales tax refund (policyengine-us #9946), the IRA deduction's active-participant phase-out (policyengine-us #10032; four outputs), New York's 2026 child and dependent care credit (policyengine-us #9948) and the IRA deduction's active-participant phase-out and estate income in gross income (policyengine-us #10032, #10027; two outputs). PolicyBench still builds each reference from the stated facts and from law published before it froze the references on 2026-07-03, under the same publication conventions.

Two rulings of October 6 apply too. PolicyBench's reference adversary flags the scored outputs where many models, or several of the strongest, agree on an answer the reference does not give, and a judge that cannot yet see the engine's derivation works each one from primary law. Its findings led to rulings to exclude eight outputs. Four were engine defects, in Arizona, Ohio, Colorado and New York; the new version fixes them, so PolicyBench keeps scoring those four instead. The other four are the federal and state income tax of a Pennsylvania and a Missouri household whose dependent earns $45,000 of wages and must file their own return. The output definitions do not say whether the household's income tax includes that return, so PolicyBench stops scoring them. It also stops scoring two Louisiana households' state income tax, whose reference rests on a 2026 standard deduction computed from published price indexes; Louisiana published the amount itself on September 28, after the freeze.

The new version counts Indiana county income tax in local income tax, and the prompt names no county, so PolicyBench stops scoring the local income tax of two Indiana households. It also counts Idaho's $10 permanent building fund tax in state income tax, which moves one household's reference from $6,818.34 to $6,828.34. Every model is now scored on 1,926 of its 1,984 requested outputs; PolicyBench excludes the other 58.

Against the previous release, the exact rate of every one of the 46 earlier models falls, by 0.24 to 1.37 points. The cause is the 14 restored outputs that the previous release did not score, which few models get right: half of them are answered within $1 by at most one of the 47 models, and four by none. Had they stayed excluded, every earlier model's rate would have risen, by 0.39 to 0.71 points.

PolicyBench stops scoring eight tax outputs, in six households, whose references turn on facts the prompts never state. The eight are federal income tax for households in California, Massachusetts, and Virginia; the Virginia household's state income tax; and payroll tax for the Massachusetts household and households in Minnesota, Colorado, and New York. Every model is now scored on 1,920 of its 1,984 requested outputs, down from 1,928, and PolicyBench excludes 64.

The three federal outputs depend on the state income tax each household paid during 2026, which the federal deduction for state and local taxes counts and no prompt lists. policyengine-us 2.15.17, which built the references, fills that amount with its own estimate of state income tax withheld, a formula on federal adjusted gross income. If each household paid its 2026 state income tax as the engine computes it, the outputs are $11,895.23 in California, $26,991.31 in Massachusetts, and $11,137.31 in Virginia, against references of $11,113.57, $26,932.27, and $10,729.61.

The Virginia household's head is 69, and the prompt lists no Medicare enrollment or premium. policyengine-us treats everyone eligible for Medicare as enrolled and counts the standard Part B premium, $2,434.80 for 2026, as a medical expense. The household itemizes, and Virginia's itemized deductions start from the federal ones, so the premium lowers its Virginia income tax: without it, that output is $3,654.15 rather than the reference's $3,514.15. The premium lowers the household's federal income tax too, and that output's exclusion record names both unstated facts.

The four payroll outputs count an employee share of a state paid-leave or disability premium: Minnesota Paid Leave, Colorado FAMLI, Massachusetts Paid Family and Medical Leave, and New York Paid Family Leave and Disability Benefits. In each of these states the premium is the employer's, and the employer may deduct up to a set share of it from wages but need not, so the law requires no amount of the employee. The prompts ask for mandatory employee state payroll taxes, and the references count the largest share each state's published 2026 rates let the employer deduct, without the prompts saying whether it does. Without that share, the outputs are $2,218.50 in Minnesota, $312.74 in Colorado, $13,387.65 in Massachusetts, and $7,664.92 in New York, against references of $2,346.10, $330.73, $14,192.66, and $8,108.03. PolicyBench still scores the payroll outputs of seven households in California, Connecticut, and Pennsylvania, where the law puts the contribution on the worker and requires the employer to withhold it.

No model's answer is within $1 of the reference on any of the four income tax outputs. On the four payroll outputs, 35 answers match the reference within $1: six for Minnesota, 16 for Colorado, nine for Massachusetts, and four for New York. Another 82 match the amount without the state share.

Every model's exact rate, its share of answers within $1 weighted by household impact, rises by 0.67 to 1.35 points. GPT-6 Sol still leads at 95.8%, ahead of Claude Opus 5.5 (95.1%) and GPT-5.6 Sol (94.4%). Claude Sonnet 5.5 (93.3%) passes GPT-6 Luna (93.0%) for fourth. Further down, Inkling moves above Grok 4.7, Gemini 3 Flash Preview above Claude Opus 4.7, and Gemini 3.1 Flash Lite Preview above DeepSeek V4 Pro. Every other pair keeps its order.

PolicyBench's next prompt contract remains an unactivated draft (PolicyEngine/policybench#173). It states the state income tax paid for each tax unit or a named convention for estimating withholding; each person's Medicare Part B enrollment and annual net premium; and whether each person's employer withholds the employee share of state paid-leave or disability premiums. Its payroll output definition counts optional employee shares only when the employer withholds them. A later run can score these outputs after its reference builder and a fresh input sweep validate the stated conventions.

Two corrections in this release move no score. PolicyBench now reports that it collected model responses from June 12 to September 29, 2026, ending on the UTC date of the last answer; release dashboard-data-20260930 gave September 30, the date of that release. The audit text for a California household head's Medicaid eligibility now explains the head's income test as the engine applies it: California's $230 monthly income disregard, which takes the place of SSI's $20 general income exclusion, brings countable income under the limit. The earlier text said the engine subtracts a Medicare Part B premium from income, which it does not.

For each of five households that qualify for SNAP food benefits, most of the 46 models on PolicyBench answer $0. SNAP pays every eligible household of one or two people at least a minimum benefit, which in these households' states is $24 a month through September 2026 and $25 from October. Each of the five has one or two people, so an answer of $0 says the household does not qualify, and that answer could keep someone who qualifies from applying. Of the 230 answers PolicyBench requests for these households, 190 come to $0 and seven match the reference.

PolicyBench grades answers against references from PolicyEngine, the open-source tax and benefit model, and the references follow law published before PolicyBench's July 3 reference freeze. USDA published the $25 minimum on August 21, so the references keep $24 through December and put each household at that minimum for each month it qualifies: $288 for 2026 for four households, and $240 for an Arizona resident who qualifies from March.

Each household qualifies only through broad-based categorical eligibility (BBCE). Ordinarily, SNAP tests a household's gross income against 130% of the federal poverty guideline, its net income after deductions for costs such as housing, and its savings. Under BBCE, a state extends SNAP to households that receive a non-cash benefit funded by Temporary Assistance for Needy Families (TANF), or that the state has authorized to receive one. Arizona, Connecticut, Michigan, and Wisconsin open that benefit, and with it SNAP, to households with gross income up to 200% of the guideline, and Texas opens it to those up to 165%. Arizona raised its limit from 185% to 200% starting with benefit month March 2026. None of these states tests net income for the non-cash benefit, and only Texas keeps an asset limit for it.

All five households have gross income above 130% of the guideline, but SNAP exempts households with an elderly or disabled member from its ordinary gross income test. Four of the five have such a member: a Connecticut couple on Social Security disability income, a disabled Michigan resident living alone, an 85-year-old surviving spouse in Wisconsin, and an 80-year-old Arizona resident on Social Security and a pension. The fifth, a 23-year-old Texas resident with wages and $12,000 of financial assistance, fails that test. The four exempt households fail SNAP's ordinary net income test instead, and the Connecticut couple and the Arizona resident also hold $48,000 and $58,700 in savings, above SNAP's asset limit. Each household's gross income is under its state's BBCE limit, the Arizona resident's from March. BBCE leaves SNAP's benefit formula in place, and at these incomes the formula pays nothing, so each household gets the minimum.

The Michigan resident's prompt lists employer-sponsored insurance premiums, and PolicyEngine treats the employer as paying them. Had the resident paid them, SNAP would count them as a medical expense, and the household would pass the net income test and get the same minimum without BBCE. GPT-5.5 gets the Michigan resident's amount right by that route: its explanation deducts medical premiums, puts net income below the net income limit, and applies the minimum.

No prompt mentions the non-cash benefit, and each mentions TANF only to ask for the household's TANF amount. Every prompt tells the model to treat any unlisted household fact or status "as false", to assume "program take-up when required", and not to infer unlisted "benefit receipt". The rules on unlisted statuses and benefit receipt tell a model the household lacks the non-cash benefit, while the take-up rule can tell it the household takes that benefit up. PolicyEngine applies BBCE to any household eligible for the non-cash benefit, without checking whether the household receives it or holds the state's authorization to receive it. A model that requires receipt or authorization falls back on SNAP's ordinary tests, and each of the five fails at least one. Among the explanations for the 190 answers of $0, TANF appears in one: Claude Sonnet 4.6 writes that Texas has no BBCE and that the Texas resident receives neither TANF nor Supplemental Security Income.

When savings rather than income hold a household back, most models answer above $0. PolicyBench also asks models about households in New Jersey, North Carolina, Virginia, and Pennsylvania that pass SNAP's income tests but hold savings above its asset limit, which BBCE removes in those states. Of the answers models give for those four households, 77% come in above $0, against 17% for the five held back by income. Of the 46 models, 12 answer above $0 for each household held back by savings and $0 for each of the five held back by income, among them GPT-5.6 Sol and GPT-6 Luna, #3 and #4 on PolicyBench. PolicyBench does not score the Pennsylvania household's SNAP amount, because PolicyEngine grants it the heat-and-eat utility allowance, which a 2025 law narrowed. With or without that allowance, the household qualifies for a benefit.

PolicyBench asks each model to explain every answer, and the explanations mention BBCE less often for the households held back by income. For the households held back by savings, 62 of 182 explanations mention BBCE, by name or as categorical eligibility, and 54 of those also mention assets. For the five held back by income, 29 of 228 explanations mention BBCE, and 13 of those still end at $0. Of those 13, six write that the household's income exceeds its state's BBCE limit: four for the Arizona resident and two for the Texas resident. PolicyEngine puts both households under their states' limits, the Arizona resident from March. Another five cite a net income limit or test, though none of these households' states applies one under BBCE. The last two, from Gemini 3 Flash Preview and Gemini 3.1 Pro Preview, write that the Connecticut couple meets BBCE, then answer $0 because the benefit formula pays nothing, and neither applies the minimum. Of the 16 that mention BBCE and answer above $0, six match the reference, and seven apply the minimum at its amount through September 2025, $23 a month, and answer $276.

GPT-6 Astra gets three of the five households right, more than any other model, and cites the state's BBCE limit of 200% of the guideline each time. It answers $0 for the Texas resident, the one household that fails SNAP's ordinary gross income test, and for the Arizona resident. GPT-6.1 Sol gets two right, the Michigan and Wisconsin households, citing each state's categorical eligibility limit. Claude Opus 5.5 answers $288 for the Wisconsin surviving spouse, citing Wisconsin's BBCE and its 200% limit. For the Connecticut couple, the disabled Michigan resident, and the Arizona resident, it answers $0, citing SNAP's ordinary net income test, which BBCE removes in all three states. GPT-6 Sol, which leads PolicyBench overall, answers $0 for four of the five and mentions BBCE in one of its SNAP explanations for PolicyBench's 100 households.

GPT-6 Sol, Claude Opus 5.5, and Claude Sonnet 5.5 miss the Texas resident's amount because they count less income than PolicyEngine does. GPT-6 Sol answers $1,208.40, and Claude Opus 5.5 and Claude Sonnet 5.5 each answer $1,208. All three compute an ordinary benefit from wages alone. PolicyEngine also counts the $12,000 of financial assistance as income, which puts the household above SNAP's ordinary limits and leaves BBCE as its only route. The prompt labels that money only "financial assistance", without naming its source, and SNAP counts some kinds of assistance as income but excludes others, such as educational assistance.

All 46 models answer $0 for the Arizona resident, whose gross income falls between Arizona's old and new BBCE limits. PolicyBench's references use policyengine-us 2.15.17, which encodes the raise. GPT-6 Astra, GPT-6.1 Sol, Gemini 3.5 Flash, and Gemini 3.8 Flash write that the resident's income exceeds Arizona's categorical eligibility limit; Gemini 3.5 Flash and Gemini 3.8 Flash, along with Gemini 3 Flash Preview, give Arizona's limit as 185%, the limit before March.

This note corrects PolicyBench's September 3 note. That note counted six households, four of them among the five here, at $287.68 each, an amount PolicyBench computed from PolicyEngine's unrounded minimum. SNAP rounds the minimum to the nearest dollar, and so does PolicyEngine (policyengine-us #9162). One of the six, a Michigan worker who pays child support, does not qualify. Michigan counts that child support in gross income and deducts it when computing net income, which puts the worker's gross income above Michigan's 200% BBCE limit; PolicyEngine subtracted it from gross income until policyengine-us #9586. The version of this note published September 23 counted the worker among its households. PolicyBench no longer scores another of the six, a second Texas household, whose reference assumed 40 hours of work a week that the prompt does not list. At 0 hours, as the prompt's rule for unlisted numbers reads, PolicyEngine gives that household $0, the answer GPT-5.6 Sol, Claude Fable 5.1, and Kimi K3 gave in the September 3 note. The September 3 note also says SNAP excludes that household's farm rent, but SNAP counts rent, farm rent included, net of the costs of doing business: as unearned income under 7 CFR 273.9(b)(2)(ii), or as earned income under 273.9(b)(1)(ii) when a household member averages at least 20 hours a week managing the property. PolicyEngine counts farm rent from policyengine-us #9671, which postdates the references' engine, 2.15.17. The September 3 note described BBCE as covering households that receive the non-cash benefit, but PolicyEngine computed that note's amounts by applying BBCE to every household eligible for that benefit. It also said Connecticut, Michigan, Texas, and Wisconsin waive the asset test, but Texas keeps a $5,000 asset limit.

A later release, dashboard-data-20261006, stops scoring eight tax outputs and changes no SNAP reference or answer. On it, GPT-6 Luna is #5 on PolicyBench, below Claude Sonnet 5.5, and every other figure in this note stays the same.

Claude Sonnet 5.5, Grok 4.7 and DeepSeek V4.1 Flash joined the board on 2026-09-29, bringing it to 45 models. Claude Sonnet 5.5 scores 92.1% of answers within $1, weighted by household impact, #5 of 45. GPT-6 Luna, #4, also rounds to 92.1% and sits about level with it, less than 0.02 points ahead. Grok 4.7 scores 88.3% (#11) and DeepSeek V4.1 Flash 87.1% (#15). GPT-6 Sol still leads at 95.0%, ahead of Claude Opus 5.5 (93.7%) and GPT-5.6 Sol (93.6%).

Claude Sonnet 5.5 costs $0.0343 a household, against $0.0019 for GPT-6 Luna. DeepSeek V4.1 Flash costs $0.0242 at DeepSeek's peak list price, and Grok 4.7 costs $0.2597.

Claude Sonnet 5.5's API rejects forced tool calls, as Claude Opus 5.5's and Claude Fable 5.1's do, so its row answers as a JSON object and reasons with adaptive thinking, the provider default. Grok 4.7 answers through the forced tool call with a 1,800-second request timeout, because its whole-household onboarding probe ran past a 600-second timeout. DeepSeek serves V4.1 Flash under the alias deepseek-flash, and all 1,984 of the row's answers report that alias and one system fingerprint; none reports a version. PolicyBench labels the row from DeepSeek's September 10 release note, which put V4.1 Flash on that alias.

The release also moves PolicyBench's scored references from policyengine-us 1.755.4 to 2.15.17, the newest release when PolicyBench began sweeping the references on 2026-09-29 (uploaded at 00:23 UTC). The newer version includes the eight upstream fixes that PolicyBench applied as sandbox fixes for the September 22 references, and it encodes law the older version lacked. PolicyBench still builds each scored reference from the stated facts and law published before it froze the references on 2026-07-03, so it ported its nine publication conventions to the new version. With those conventions and an adapter that keeps Maryland county tax out of state income tax, policyengine-us 2.17.0, the newest release when PolicyBench checked PyPI on 2026-09-29 at 14:58 UTC, gives the same value as 2.15.17 for all 1,984 outputs.

The move changes four scored references. New Jersey's child tax credit schedule for 2026 to 2028, which the state approved on June 30, raises one household's state refundable credits from $5,342.40 to $5,842.40. Arizona raised the income limit for its broad-based categorical eligibility from 185% to 200% of the poverty guideline effective March, and the new limit gives an Arizona household $240 of SNAP where the reference was $0. policyengine-us now counts child support received as school-meal income, which ends a Pennsylvania household's eligibility for reduced-price meals. A fix to the rounding in New York's Empire State child credit phase-out (policyengine-us #9425) raises a New York household's state refundable credits from $650.50 to $667.

PolicyBench stops scoring three federal income tax outputs. policyengine-us 2.15.17 counts the whole of a listed state and local tax refund as income. Federal law counts the refund only to the extent the refunded tax lowered the household's federal tax in the year the household paid it, and the prompts do not say whether it did. PolicyBench also stops scoring one California household head's Medicaid eligibility, which its re-review of the household's excluded SNAP output flagged. On disability, the prompt says only that the head is disabled. The head's income, 141% of the poverty guideline, is above the 138% limit for the adult expansion group, so only a disability pathway leads to Medi-Cal, California's Medicaid program. Medi-Cal's Working Disabled Program requires SSI's definition of disability, which the prompt does not state, and policyengine-us 2.15.17 tests the general disability flag instead. The same unstated fact already keeps the household's SNAP out of scoring. PolicyBench now scores every model on 1,928 of its 1,984 requested outputs and excludes 56. Another two references move by less than $1, and PolicyBench re-reviewed the 19 excluded outputs whose values moved; all stay excluded.

Together, the new references and the Medicaid exclusion raise the exact rate of every one of the 42 earlier models, by 0.14 to 0.83 points. None of the 42 matched any of the three federal outputs now excluded, and all 42 had matched the Arizona household's old $0 SNAP reference, which none matches now. On the Medicaid output, 20 of the 42 had matched the reference and 22 had not. Among the 42, five models move up in the order: GLM-5.3-Flash (preview) moves above Inkling, Claude Opus 5 above Claude Fable 5, DeepSeek V4 Pro 0813 above DeepSeek V4 Flash 0731 and Gemini 3.7 Flash, Grok Build 0.1 above Claude Sonnet 4.6 and Gemini 3.5 Flash, and DeepSeek V4 Pro above Gemini 3.1 Flash Lite Preview. Every other pair keeps its order.

The Arizona household's income keeps it from qualifying for SNAP under the program's ordinary tests, so it qualifies only through broad-based categorical eligibility, and all 45 models answer $0 for it. PolicyBench's October 5 note on such households counts it among five held back by income, with four in Connecticut, Texas, Michigan and Wisconsin. That note's data list every model's answer for those five households and for the four held back by savings, including the three models this release adds.

GPT-6 Sol, GPT-6 Luna and Claude Opus 5.5 joined the board on 2026-09-22. GPT-6 Sol leads at 94.2% of answers within $1, weighted by household impact, #1 of 42 and 1.3 points above Claude Opus 5.5 (92.9%, #2). GPT-5.6 Sol, the previous leader, is #3 at 92.7%.

GPT-6 Luna scores 91.4% (#4) at $0.0019 a household, against $0.0271 for GPT-6 Sol and $0.0675 for Claude Opus 5.5.

Claude Opus 5.5's API rejects forced tool calls, as Claude Fable 5.1's does, so its row answers as a JSON object and reasons at the provider default. Claude Opus 5's board row runs the forced tool call, which switches that model's thinking off, and scores 83.4%; its tool_choice auto re-run, where it reasons, scores 89.3%. The two Opus rows differ in serving shape as well as in model, so the 9.6-point gap between them is not a measure of the model change alone.

Every row is scored on 1,932 of its 1,984 requested outputs. This release regenerated 26 references and excludes 52 outputs from scoring for every model; the reference audit note explains both.

The next board version moves every model to tool_choice auto so each provider's default reasoning engages under the recorded request shape (policybench#139).

A later release, dashboard-data-20260922c, corrects the reference for one output: the SNAP amount of a Michigan worker who pays child support. PolicyEngine had subtracted that child support from the worker's gross income; with its fix (policyengine-us #9586) the worker does not qualify, and the reference is $0 instead of $288. The later release scores the same outputs, regenerates the same references and excludes the same outputs as this one. On it, GPT-6 Sol scores 94.3% and Claude Opus 5.5 93.0%, and Claude Opus 5's tool_choice auto re-run scores 89.4%. The note's other scores, gaps, ranks and costs stay the same. An interim release, dashboard-data-20260922b, excluded that output while the fix was open.

Claude Opus 5.5 judged the misses in the cases the September 22 additions joined, and it flagged 39 references as suspect. Developers settled each flag against statutes, regulations, agency publications and runs of the reference engine. The adjudication affirmed 5, replaced 6 with a regenerated reference, traced 8 to an input the prompt never states, and confirmed 20 as defects in policyengine-us 1.755.4 that are not yet fixed upstream.

Flags sample defects without bounding them. For each defect we implemented the rule as a fix on the same engine version and recomputed all 1,984 references under it. 7 of the root causes had been fixed in policyengine-us after the references were frozen, so the 15 scored references they move are regenerated with the fix. One of those fixes as first built went further than the upstream fix, applying Wisconsin's capital gain exclusion to capital gain distributions; upstream still leaves them out, so that part counts as a defect not yet fixed. Every output a defect not yet fixed upstream moves by more than a dollar is excluded from scoring for every model: 28 outputs in 20 households are recorded as engine-defect exclusions across 11 root causes, and 8 of them had never been flagged. An output such a defect moves that was already excluded for an unstated input keeps that record. Also excluded are 24 outputs whose reference turns on an input the prompt never states, such as who paid for the coverage behind a disability benefit.

The audit also tested a failure no judge flagged. policyengine-us extends a parameter past its last encoded year with a price-index projection, so a 2026 reference can rest on a forecast where a government published an amount. Recomputing every reference with each projection held found forecasts behind references in SNAP, California, the IRS sales tax tables, Wisconsin, Minnesota, Michigan, Missouri, Maryland and Idaho. The rule the benchmark now states: a scored reference follows from the household facts and from law published before the references were frozen on 2026-07-03. The scored references among them were regenerated with the published amounts, or with the last ones published before the freeze; Wisconsin's published amounts reach only the corrected values of Wisconsin outputs excluded for engine defects.

Separating the SNAP convention's parameter holds from its arithmetic exposed engine defects in SNAP itself: the engine kept cents in the monthly allotment where federal rules require whole dollars, set the minimum benefit at $23.84 a month where the rule rounds it to $24, and rounded net income down. policyengine-us fixed each after the freeze, so the SNAP references apply those fixes. One SNAP defect is not fixed upstream: the engine grants the heat-and-eat utility allowance to households without an elderly or disabled member, which a July 2025 law ended, and the scored SNAP output it moves is excluded.

In all, 26 references are regenerated, 13 of them SNAP, and every model is scored on 1,932 of its 1,984 requested outputs. The exclusion record, the adjudications and the reference sidecar, which lists each regenerated value with its source, ship with the release.

A later release, dashboard-data-20260922c, corrects the reference for one output: the SNAP amount of a Michigan worker who pays child support. policyengine-us subtracted that child support from gross income, while Michigan counts it in gross income and deducts it only when computing net income. PolicyEngine fixed this after the audit (policyengine-us #9586), so the later release regenerates the output with the fix: counted in gross income, the worker's income exceeds Michigan's broad-based categorical eligibility limit, and the reference is $0 instead of $288. That makes 8 root causes fixed upstream; every other count in this note stays the same. An interim release, dashboard-data-20260922b, excluded that output while the fix was open.

GPT-6 Astra reached the API on 2026-09-04 and joined the board on 2026-09-05 at 88.0% of answers within $1, #2 of 39, behind GPT-5.6 Sol (89.2%) and ahead of Claude Fable 5.1 (86.9%, #3). Every row on this board is scored on 1,973 of its 1,984 requested outputs: 11 outputs whose reference depends on an engine input the household facts never list are excluded for every model.

Row by row, the two OpenAI models agree far more than they differ. Both are right on 1,816 scored outputs and both wrong on 95. The gap is 33 outputs Astra misses and Sol gets right, against 29 the other way.

Astra's solo misses cluster in two invented rules. In 13 rows it treats a listed employer-sponsored insurance premium as a pre-tax salary reduction and subtracts it from wages before computing payroll tax, federal income tax or state income tax; the household facts state gross wages, and the reference keeps them. In 10 rows it marks a household member under 65 who is flagged disabled as Medicare-eligible; the reference grants Medicare before 65 only with 24 months of Social Security disability receipt recorded, and none of these households carry that input. The remaining 10 solo misses are one-off rule errors with no shared mechanism. One further Medicare row of the same kind, on scenario_074, sits on an excluded output and is outside every count here.

The judge's row notes name an employer-sponsored insurance premium on 26 of Astra's annotated rows and on 4 of Sol's. Sol netted the same premium once, on scenario_045, which cost it 2 of its 29 solo misses; the rest of Sol's solo misses share no mechanism.

The row list, with both predictions, the reference and the judge's note for every disagreement, is committed beside this note. Click any scenario in the explorer to see the household facts and each model's answer.

20 of the 100 households qualify for SNAP in the PolicyEngine reference. The top three models on the board, GPT-5.6 Sol, Claude Fable 5.1, and Kimi K3, each predict $0 for the same 6: scenario_027, scenario_030, scenario_045, scenario_073, scenario_108, scenario_112. The reference for each is $288¹ for the year, the minimum allotment (8% of the one-person maximum, $23.84 a month, for households of one or two).

Five of the six qualify through broad-based categorical eligibility: Connecticut, Michigan, Texas, and Wisconsin confer SNAP eligibility on households receiving a TANF-funded non-cash benefit, with gross-income limits of 200% of poverty (CT, MI, WI) and 165% (TX) and the net-income and asset tests waived. The sixth, scenario_112 in Texas, passes the federal tests; the models counted farm-rent income the rules exclude.

10 of the 20 eligible households qualify only through categorical eligibility: 6 because income exceeds the federal limits (one of those also exceeds the asset limit), 4 because savings alone exceed the federal asset limit. On the 4 asset cases the models compute benefits near the reference: Sol within 1% on three and 10% on one, Fable 5.1 within 1% on all four.

Across the 100 households, Sol's SNAP explanations mention categorical eligibility in 1 and assets in 36; Fable 5.1's mention broad-based categorical eligibility in 28; Kimi K3's mention categorical eligibility in 17.

Pathways were recomputed with policyengine-us 1.723.0 from the frozen scenarios and agree with the frozen reference SNAP values for all 100 households; the reference itself was generated with policyengine-us 1.755.4.

A later note, published October 5, corrects this one. From release dashboard-data-20260922c on, PolicyBench scores the SNAP amounts of four of these six households at $288 each, and scores a Michigan worker who pays child support at $0: PolicyEngine counts that child support in gross income, as Michigan does (policyengine-us #9586), and the worker does not qualify. PolicyBench no longer scores a Texas household's SNAP amount, which PolicyEngine computed with hours of work the prompt does not list.

Claude Fable 5.1 was released on 2026-09-01 and joined the board the same day. Board row: 86.3% of answers within $1, #2 of 33. 1,984 of 1,984 answers parsed.

The model's API rejects forced tool calls, which is how the board collects most models' answers, so its row answers as a JSON object and reasons at the provider default. The serving-configuration table in the paper records the transport for every row.

Sensitivity: with the answer tool declared under tool_choice auto, Fable 5.1 scores 87.5% (would rank #2). Under the same request Claude Fable 5 scores 86.9%. Fable 5's board row (79.9%) ran chunked under a forced tool call, which switches off its thinking, so the 6.4-point gap between the two board rows is mostly request shape.

The next board version moves every model to tool_choice auto so each provider's default reasoning engages under the recorded request shape (policybench#139).