A research team led by Prof. DING Zenghui from the Hefei Institutes of Physical Science of the Chinese Academy of Sciences, together with collaborators from the University of California and other institutions, has developed new approaches to improve the reliability of reasoning in large language models (LLMs).
The team explored reinforcement learning strategies that can improve LLM reasoning and proposed a new benchmark system, Prism Benchmark OPG-D3, for evaluating AI model generalization.
The related studies have been accepted by ACL 2026.
Current reinforcement learning approaches for LLMs mainly optimize models based on whether the final answer is correct. However, this may cause models to learn superficial patterns rather than develop reliable reasoning processes.
Inspired by cognitive control mechanisms in the human brain, the researchers found that Process Reward Models (PRMs), which provide feedback on intermediate reasoning steps, can better guide models to refine their reasoning and reduce shortcut learning compared with Outcome Reward Models (ORMs), which only evaluate final results.
The study further shows that simply increasing training data is not sufficient to overcome these limitations, while structured supervision of reasoning processes is important for improving model reliability.
The team also introduced Prism Benchmark OPG-D3, an evaluation framework designed to test whether reinforcement learning models can generalize beyond familiar tasks. The framework includes the Oracle Performance Gap (OPG) metric, which measures the difference between a model trained on regular data and one optimized directly on test data.
Experiments on widely used benchmarks showed that current evaluation methods may underestimate generalization risks. Prism Benchmark OPG-D³ provides a more comprehensive framework for assessing model reliability under challenging and unfamiliar scenarios.
The findings provide new insights into how LLMs learn and reason, highlighting the importance of improved training strategies and evaluation methods for developing more reliable Artificial Intelligence systems.
Illustration of outcome reward-induced shortcuts and the corrective mechanism of process supervision (Image by DING Zenghui)
