SoMBench asks AI to read the room. The room is still staged

Cheng Xueqi, deputy director of the Institute of Computing Technology, Chinese Academy of Sciences, speaks at the July 10, 2026 Sugon 8000 launch where Zhijing was presented. The photograph records the event, not the benchmark's annotation or evaluation process.[2]

A language model can pass a false-belief puzzle and still mishandle a room. It may know that one person has not seen an object move, yet miss why a request is indirect, which colleague has authority, when a polite answer becomes evasive, or how a rule changes when roles and relationships change.

SoMBench is an attempt to make that larger room measurable. The Chinese Academy of Sciences' Zhijing team presented the benchmark publicly in July 2026 as the evaluation layer of a broader “social mind” program. Instead of reducing social cognition to one theory-of-mind score, SoMBench divides it among mental-state inference, strategic interaction, and the interpretation of norms. Its technical report evaluates 20 models on 3,481 Chinese-language items built from 284 scenarios.[1][2]

As of August 8, 2026, the benchmark's strongest contribution is not the headline that its best tested model scored 72.08%. It is the diagnostic structure underneath that number: the same social setting can be probed through constrained and open answers, first- and third-person viewpoints, short and long contexts, and fine-grained capability labels. The corresponding boundary is just as important. These are deliberately staged written situations, not observed conduct in a changing social environment, and the public release is not yet sufficient to reproduce the reported table end to end.[1][3]

Image context: the cover is an official event photograph of Cheng Xueqi speaking at the launch where Zhijing was presented. It establishes the institutional and public setting of the project; it does not depict the creation of SoMBench or independently verify its results.[2]

From one false belief to a whole social field

Earlier theory-of-mind benchmarks made an essential problem tractable: can a model distinguish what happened from what a particular person knows or believes happened? ToMBench, for example, organized eight task families and 31 abilities into a bilingual multiple-choice evaluation, explicitly using a newly constructed inventory to reduce contamination and subjective scoring.[5]

SoMBench expands the target in two directions. Its first primary dimension, mentalizing, includes beliefs, desires, intentions, emotions, perspective, empathy, and personality or preference. Its second, strategic navigation in social interaction, covers communication, negotiation, bargaining, cooperation and conflict, trust and deception, and group dynamics. Its third, internalization and dynamic balance of social norms, covers norms, identity and role, power and institutions, and group boundaries. Those three branches contain 17 secondary dimensions and 71 task paradigms.[1]

That taxonomy matters because “social intelligence” is otherwise too elastic to falsify. A model can be good at recognizing emotion and bad at separating belief from fact. It can produce tactful language while failing to notice that a subordinate cannot safely refuse. It can recite a norm while applying it without regard to role, history, or conflicting obligations. A single average hides those different failure modes; a labeled grid makes them inspectable.

The expansion also changes what a pass means. SoMBench is wholly Chinese in the reported evaluation, which is a useful correction to an English-heavy benchmark landscape. It tests whether current models can work through social cues expressed in Chinese rather than treating translated English conventions as universal. But language coverage is not cultural validity. A Chinese scenario can still encode one authorial view of a contested norm, and a correct reference answer does not turn that view into a universal social law.[1]

The construction pipeline is the real technical result

The team did more than prompt a model for difficult questions and publish the output. It began with capability-specific scenario requirements, created controlled rewrites, used model failures to select harder variants, and then subjected every candidate to human review. Fifteen undergraduate and graduate reviewers with psychology backgrounds worked through a custom annotation system. Two people independently reviewed each instance; a third resolved disagreements. Reviewers first checked scenario coherence and knowledge boundaries, then solved questions without seeing the reference answer before inspecting the full question-and-answer chain.[1]

Of 3,484 generated candidates, 3,034 initially passed. Three were discarded because they could not be repaired; the other non-passing cases were revised and checked again, producing the final 3,481 expert-verified instances. The set contains 1,704 original items and 1,777 rewritten or harder variants. It is split among 713 single-choice questions, 626 multi-select questions, 1,661 judgment items, and 481 open-ended analyses.[1]

Those details are more informative than the raw item count. Controlled rewrites can reveal whether a model follows a character's information boundary or merely recognizes a familiar story template. First-person and third-person versions can expose perspective leakage. Multi-select questions can test whether the model maintains several compatible constraints instead of grabbing the first plausible answer.

There is precedent for treating such perturbations as evidence rather than decoration. A 2024 Nature Human Behaviour study compared models and 1,907 human participants across multiple theory-of-mind tasks, then used altered items to show that apparently strong or weak performance could reflect response bias and prompt sensitivity rather than a stable human-like faculty.[4] SoMBench's variants pursue the same methodological instinct at a broader scale: do not trust a polished answer until the setup changes.

The leaderboard hides the sharper signal

The 20-model table has an obvious headline. Claude Opus 4.8 led at 72.08%, followed by GPT-5.4 at 69.35% and GPT-5.5 at 69.09%. The weakest tested model scored 42.69%, leaving a 29.39-point spread. Across the pool, no model reached the report's 90% “near-ceiling” threshold on any of the 17 secondary dimensions.[1]

Yet the format breakdown says more about the evaluation problem. Averaged across models, single-choice accuracy was 94.47%, while multi-select accuracy was 55.60%, judgment accuracy was 64.06%, and the open-ended pass rate was 62.86%. A model can look socially fluent when one attractive answer is placed among distractors and much less reliable when it must preserve every applicable belief, relationship, or norm.[1]

That gap should not be read as a clean causal estimate of “format difficulty.” The question types ask different things, and the open responses use a different scoring mechanism. Even within shared scenarios, changing from selection to analysis changes both the output surface and the cognitive demand. The result establishes setup sensitivity, not a universal 38.87-point penalty for removing radio buttons.

Perspective and length are similarly non-monotonic. Third-person questions were about two points easier on average than first-person questions, but some dimensions reversed that direction. Long contexts were only 0.12 points easier on average, with role and bargaining benefiting while intention and empathy favored shorter contexts. More text did not reliably supply more usable social understanding; sometimes it supplied more relationships and constraints to lose.[1]

This is why the aggregate should not be marketed as a social IQ. It mixes distinct abilities, four answer formats, and two scoring routes. Its best use is as a matrix: identify where a model's result changes when perspective, context, or response obligations change, then inspect the cases.

Open answers import a second model into the score

Closed questions are matched against reference labels. The 481 open-ended items are different: two automated judges assign continuous rubric scores, and a fixed threshold converts those scores into pass or fail. The reported open-answer number therefore measures a three-part system—the tested model, the rubric, and the judge models—not the tested model alone.[1]

That is a legitimate way to evaluate explanation quality, but the receipt has to name every component. The technical report describes two judges without identifying their model snapshots in the benchmark section or stating the numeric threshold there. The public SoMEval repository currently provides the missing mechanics: dimension-specific rubrics, a 7-of-10 pass threshold, one active qwen3.7-max judge configuration, and an optional second-judge block that is commented out. The repository itself says a second judge can be enabled to mirror the two-judge protocol.[3]

The code is useful, but its defaults should not be silently substituted for the unpublished settings behind the paper's table. An exact rerun needs the two original judge identities and versions, their prompts and sampling settings, the threshold, the tested-model API snapshots, and the item-level results. Without that manifest, a later evaluator can reproduce the shape of the pipeline while still producing a materially different score.

Public code is not yet a public test

The release has a second reproducibility gap. At the pinned public commit inspected for this article, SoMEval contains runners, prompt builders, scoring code, rubric files, and documentation, but not the 3,481 SoMBench records. Its download script points to a Hugging Face dataset endpoint that required authorization in an unauthenticated check on August 8. Repository documentation also describes the SoMBench source as internal and notes that several reviewed base files are local, ignored inputs.[3]

A private test set is not inherently a flaw. Keeping questions out of model training can protect a benchmark from becoming another memorized public exam. But privacy needs a complementary verification path: a hosted evaluator, cryptographic dataset version, immutable model and judge manifests, downloadable item-level scores, and a process for auditing disputed labels. At present, outside readers can inspect the apparatus but cannot independently confirm that the released apparatus, hidden items, and published leaderboard are the same experiment.

The distinction is especially important because the benchmark is already feeding a training story. The Zhijing report uses capability diagnoses to guide post-training of its Zing model family, then presents an Actio inference harness that routes skills, mental-state representations, experience, and retrieved normative knowledge. That evaluation-to-training-to-deployment loop is the project's strategic idea.[1][2] If the evaluation layer is not independently inspectable, improvements inside the loop are harder to separate from specialization to its labels and rubrics.

What SoMBench proves—and what it should prove next

SoMBench proves that social evaluation can be made wider and more diagnostic than a shelf of isolated false-belief questions. It supplies a serious Chinese-language taxonomy, a documented human-review pipeline, multiple response formats, and controlled axes for perspective and context. Its own results show that model rankings and apparent competence move with those choices.[1]

It does not prove that a 72.08% system understands people in an open environment. The scenarios remain controlled; open answers depend on automated judges; cultural and institutional norms can shift; and the public artifacts do not yet support an exact external rerun. The report acknowledges the conceptual version of this limit: real social intelligence combines several cues over time and must handle ambiguity, conflict, and changing contexts rather than only isolated labeled capabilities.[1]

Four next receipts would turn a promising benchmark into durable infrastructure:

A verifiable evaluation route: release the versioned test set or provide a hosted runner with item-level audit records.
A complete judge manifest: freeze both judge snapshots, rubrics, threshold, prompts, and inference settings for every reported table.
Human and cultural baselines: measure agreement, disagreement, and calibration across populations instead of treating every norm label as singular.
Behavior beyond answers: test multi-turn repair, uncertainty, deferral, and action under changing relationships, not only responses to completed stories.

The project's name, Zhijing, can be translated loosely as knowing the situation. SoMBench's most honest finding is that models do not know a situation independently of how it is presented. Change the viewpoint, the amount of context, the answer format, or the judge, and the apparent social mind changes with it. That is not a reason to dismiss the benchmark. It is the reason to preserve every part of the room.

cronfeed.work