Given a tool that runs PolicyEngine, three models score above 99%
PolicyBench gave three models a tool that runs PolicyEngine on a household the model writes from the prompt's facts, and ran its 100 US households again on October 14 and 15. GPT-6 Sol, which leads the board at 95.2% of answers within $1, weighted by household impact, gets 99.2% with the tool. Claude Opus 5.5 goes from 94.1% to 99.8%, and Gemini 4 Argon from 88.7% to 99.9%. The runs are a pilot, not board rows, since the board scores models answering alone, and the result is circular: the tool runs the engine that computes PolicyBench's references, so the scores measure agreement with that engine, not whether it is right.
Most of the answers the tool fixes are dollar amounts that are not zero. Of the 187 nonzero amounts, GPT-6 Sol gets 67.4% within $1 on its own and 96.8% with the tool; Claude Opus 5.5, 70.1% and 98.9%; and Gemini 4 Argon, 46.5% and 99.5%.
With the tool, 12 answers are wrong across the three runs: nine of GPT-6 Sol's, two of Claude Opus 5.5's and one of Gemini 4 Argon's. Each traces to the household the model described to the engine, or to the output it read back, rather than to the calculation: in seven the model had entered a stated fact under another variable, in three it had left one out, and in two it read back the wrong output. Prompt labels share the blame for two: "employer sponsored insurance premiums" never says the employer paid them, and "disability benefits" never says they are not Social Security. In five of GPT-6 Sol's, it had used all eight calculation turns the pilot allows. GPT-6 Sol is also the only model that turned right answers wrong, two of them.
The tool costs more. For the 100 households, GPT-6 Sol costs $5.63 with it and $2.71 without, Claude Opus 5.5 $7.99 and $6.75, and Gemini 4 Argon $8.01 and $2.79.
The tool applies the conventions PolicyBench uses for its references, so a model that describes a household exactly as the reference does gets the reference's answer. Describing a household is also easier here than it would be from a filer's documents: 1,065 of the 1,632 facts the prompts state, not counting ages and wages, carry a label that is the engine's own variable name with spaces.
Each model's answers with the tool, its calculations and cost per household, the remaining misses with the inputs behind them, and the script that scores them the board's way are in the data below.