Large language models (LLMs) have achieved remarkable performance in generating human-like text and solving reasoning tasks of moderate complexity, such as question-answering and mathematical problem-solving. However, their capabilities in tasks requiring deeper cognitive skills, such as common-sense understanding and abstract reasoning, remain under-explored. In this paper, we systematically evaluate abstract common-sense reasoning in LLMs using the ConceptNet knowledge graph. We propose two prompting approaches: instruct prompting, where models predict plausible semantic relationships based on provided definitions, and few-shot prompting, where models identify relations using examples as guidance. Our experiments with the gpt-4o-mini model show that in instruct prompting, consistent performance is obtained when ranking multiple relations but with substantial decline when the model is restricted to predicting only one relation. In few-shot prompting, the model's accuracy improves significantly when selecting from five relations rather than the full set, although with notable bias toward certain relations. These results suggest significant gaps still, even in commercially used LLMs' abstract common-sense reasoning abilities, compared to human-level understanding. However, the findings also highlight the promise of careful prompt engineering, based on selective retrieval, for obtaining better performance.

本研究针对大语言模型在常识理解和抽象推理方面的能力不足进行了系统评估，填补了这一研究空白。我们提出了两种提示方法，并通过实验发现，语言模型在使用示例指导选择关系时准确性显著提高，但当被限制预测单一关系时表现明显下降。这些结果表明尽管当前模型在常识推理方面仍存在显著差距，但通过精心设计提示方法，能够获得更佳的性能表现。

探索语义关系：语言模型在常识推理中的挑战