A stronger judge audits the references

Claude Opus 5.5 judged the misses in the cases the September 22 additions joined, and it flagged 39 references as suspect. Developers settled each flag against statutes, regulations, agency publications and runs of the reference engine. The adjudication affirmed 5, replaced 6 with a regenerated reference, traced 8 to an input the prompt never states, and confirmed 20 as defects in policyengine-us 1.755.4 that are not yet fixed upstream.

Flags sample defects without bounding them. For each defect we implemented the rule as a fix on the same engine version and recomputed all 1,984 references under it. 7 of the root causes had been fixed in policyengine-us after the references were frozen, so the 15 scored references they move are regenerated with the fix. One of those fixes as first built went further than the upstream fix, applying Wisconsin's capital gain exclusion to capital gain distributions; upstream still leaves them out, so that part counts as a defect not yet fixed. Every output a defect not yet fixed upstream moves by more than a dollar is excluded from scoring for every model: 28 outputs in 20 households are recorded as engine-defect exclusions across 11 root causes, and 8 of them had never been flagged. An output such a defect moves that was already excluded for an unstated input keeps that record. Also excluded are 24 outputs whose reference turns on an input the prompt never states, such as who paid for the coverage behind a disability benefit.

The audit also tested a failure no judge flagged. policyengine-us extends a parameter past its last encoded year with a price-index projection, so a 2026 reference can rest on a forecast where a government published an amount. Recomputing every reference with each projection held found forecasts behind references in SNAP, California, the IRS sales tax tables, Wisconsin, Minnesota, Michigan, Missouri, Maryland and Idaho. The rule the benchmark now states: a scored reference follows from the household facts and from law published before the references were frozen on 2026-07-03. The scored references among them were regenerated with the published amounts, or with the last ones published before the freeze; Wisconsin's published amounts reach only the corrected values of Wisconsin outputs excluded for engine defects.

Separating the SNAP convention's parameter holds from its arithmetic exposed engine defects in SNAP itself: the engine kept cents in the monthly allotment where federal rules require whole dollars, set the minimum benefit at $23.84 a month where the rule rounds it to $24, and rounded net income down. policyengine-us fixed each after the freeze, so the SNAP references apply those fixes. One SNAP defect is not fixed upstream: the engine grants the heat-and-eat utility allowance to households without an elderly or disabled member, which a July 2025 law ended, and the scored SNAP output it moves is excluded.

In all, 26 references are regenerated, 13 of them SNAP, and every model is scored on 1,932 of its 1,984 requested outputs. The exclusion record, the adjudications and the reference sidecar, which lists each regenerated value with its source, ship with the release.