We introduce CHARM, the first benchmark for comprehensively and in-depth evaluating the commonsense reasoning ability of large language models (LLMs) in Chinese, which covers both globally known and Chinese-specific commonsense. We evaluated 7 English and 12 Chinese-oriented LLMs on CHARM, employing 5 representative prompt strategies for improving LLMs' reasoning ability, such as Chain-of-Thought. Our findings indicate that the LLM's language orientation and the task's domain influence the effectiveness of the prompt strategy, which enriches previous research findings. We built closely-interconnected reasoning and memorization tasks, and found that some LLMs struggle with memorizing Chinese commonsense, affecting their reasoning ability, while others show differences in reasoning despite similar memorization performance. We also evaluated the LLMs' memorization-independent reasoning abilities and analyzed the typical errors. Our study precisely identified the LLMs' strengths and weaknesses, providing the clear direction for optimization. It can also serve as a reference for studies in other fields. We will release CHARM at https://github.com/opendatalab/CHARM .

我们介绍了CHARM，这是首个全面深入地评估大型语言模型（LLMs）在中文常识推理能力方面的基准。我们利用5个代表性的提示策略对CHARM进行评估，发现语言模型的语言定位和任务领域会影响提示策略的效果，这丰富了以前的研究结果。我们构建了紧密相连的推理和记忆任务，发现一些语言模型在记忆中文常识方面遇到困难，影响其推理能力，而其他模型在相似的记忆表现下呈现不同的推理能力。我们还评估了语言模型在无记忆推理能力上的表现，并分析了典型错误。我们的研究准确地确定了语言模型的优势和劣势，并为优化提供了明确的方向。它还可以为其他领域的研究提供参考。我们将在此URL发布CHARM。

基于 LLMs 的中国常识推理基准评估：从中国特定性到推理记忆相关性