A coefficient changes from four to two, and a physics answer loses its claim to correctness. But how much has the solver understood? PHYBench's scoring demonstration makes that question concrete. Its reference expression is 2mg + 4mv₀²/l; the candidate keeps the same terms but halves the second coefficient. The released example awards the candidate a rounded score of 47, against 100 for the matching answer.[4]

That gap is the subject of this benchmark: how to make an incorrect answer informative without quietly declaring it correct. Developed by a team centered on Peking University physics undergraduates, PHYBench became a NeurIPS 2025 paper. The university's September 22, 2025 announcement places it within a broader effort to bring physics expertise into AI education and evaluation.[1]

Giving a near miss a shape

PHYBench contains 500 original physics problems. Its dataset card describes tasks spanning mechanics, electromagnetism, optics, thermodynamics, and other fields, with solutions requiring multiple steps. The requested final product is tightly specified: one symbolic expression, allowing equivalent forms while excluding numerical approximations and chains of equations.[3] That makes the answer something a mathematical evaluator can inspect.

The scoring code first attempts to recognize equivalence through symbolic simplification and equality checks. Expressions that look different on the page can therefore receive full credit. For remaining cases, the evaluator represents numbers, variables, operations, and functions as nodes in a tree, then measures the edits needed to reconcile the structures.[4]

Think of the tree as the grammar of the formula. Multiplication connects one set of quantities; addition joins terms; an exponent changes how a variable participates. A misplaced coefficient and a missing term leave different traces in that grammar. The metric can preserve those differences after ordinary right-or-wrong grading has flattened both answers into failure.

The default scoring rule also preserves a conspicuous reward for correctness. Zero edit distance receives 100. Otherwise, the code takes 60 minus 100 times the relative distance, stopping at zero. A relative distance of 0.1 would therefore score 50; this is an illustrative calculation from the rule, not a measured model result.[4]

The consequence is easy to miss: an individual answer with a small but nonzero distance cannot collect 99 points. It sits below 60 until it clears the correctness boundary. An average score can still fall between 60 and 100 because it combines different answers. Reading an average as the percentage of problems solved would discard the distinction the metric was built to preserve.

A zero needs a second look

The implementation has another revealing feature. An empty candidate, certain unsupported expressions containing integrals or summations, and a candidate string more than three times the reference's length can receive zero before tree comparison. Conversion failures also return zero in the ordinary execution path. The documentation offers a debugging mode for surfacing errors.[4]

These rules make the evaluation boundary tangible. A model must both solve the problem and deliver an expression the released evaluator accepts. Imagine two unsuccessful submissions: one contains a cleanly formatted but wrong formula; another contains an answer buried in a format the parser cannot convert. Their identical score would conceal different repair jobs.

For a fresh comparison, I would retain the extracted expressions and inspect zero-score cases separately. That would reveal whether an improvement came from better physics, better compliance with the answer format, or both. Such an audit would add information to the score without changing its definition.

Structural closeness has a deeper limit, too. The paper explicitly says EED does not assess the entire derivation and lacks a dimensional-consistency check.[2] A formula can resemble the reference while an intermediate argument is unsound. Conversely, a small symbolic alteration can carry a large physical consequence. Partial credit is evidence about the submitted expression; interpreting it as a complete account of understanding would demand more evidence.

Keep the result attached to its experiment

In the May 18, 2025 paper revision, Gemini 2.5 Pro leads the reported model comparison with 36.9% accuracy, against a 61.9% human baseline. The human group comprises 81 Peking University physics students, including 50 Chinese Physics Olympiad gold medalists. This is a specialized comparison, not a population-wide estimate of human ability.[2]

The setup matters alongside those percentages. API evaluations used service defaults; locally evaluated models used temperature 0.6, top-p 0.95, and a 32,768-token ceiling on four 80 GB A100 GPUs. The evaluator extracted the final boxed expression.[2] Those conditions belong to the historical experiment. They do not establish a ranking of models available in October 2026.

The release also has a practical boundary. The dataset card distinguishes 100 problems with complete solutions and reference answers from 400 additional question-only examples, alongside a combined file of all 500 questions.[3] Downloading the whole question collection therefore does not, by itself, provide every reference answer needed for a complete local scoring run. A reproduction should say which file and subset it actually graded.

My reading is that PHYBench's lasting contribution lies in making grading choices visible. Correctness, symbolic resemblance, and the quality of a derivation are related judgments, but each requires its own evidence. The coefficient that changed from four to two is still wrong. The useful advance is being able to examine what survived the mistake—and to know which parts of the solution the grade never inspected.

Sources

1. Peking University School of Physics, “PKU's PHYBench physics AI benchmark accepted at a leading international AI conference” (September 22, 2025; Chinese first-hand account of the team and educational context).
2. Shi Qiu and colleagues, “PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models,” arXiv version 2 (May 18, 2025; experiment setup, human baseline, main results, and scoring limitations).
3. Eureka Lab, PHYBench official dataset card (release files, problem scope, and answer requirements; inspected October 28, 2026).
4. PHYBench team, official EED implementation and documentation (EED.py and readme.md: example scores, expression handling, scoring function, and error behavior; inspected October 28, 2026).
5. Europejska, “School of Physics” (photograph, May 17, 2020; Peking University building, Wikimedia Commons, CC BY-SA 4.0).