Recent advancements in natural language processing by large language models
(LLMs), such as GPT-4, have been suggested to approach Artificial General
Intelligence. And yet, it is still under dispute whether LLMs possess similar
reasoning abilities to humans. This study evaluates GPT-4 and various other
LLMs in judging the profoundness of mundane, motivational, and pseudo-profound
statements. We found a significant statement-to-statement correlation between
the LLMs and humans, irrespective of the type of statements and the prompting
technique used. However, LLMs systematically overestimate the profoundness of
nonsensical statements, with the exception of Tk-instruct, which uniquely
underestimates the profoundness of statements. Only few-shot learning prompts,
as opposed to chain-of-thought prompting, draw LLMs ratings closer to humans.
Furthermore, this work provides insights into the potential biases induced by
Reinforcement Learning from Human Feedback (RLHF), inducing an increase in the
bias to overestimate the profoundness of statements.

近期，大型语言模型（LLMs），如 GPT-4，在自然语言处理方面取得了显著进展并接近人工通用智能。然而，本研究对 GPT-4 和其他 LLMs 进行评估，评估了它们在判断平凡、动机和伪深的陈述的能力。我们发现，无论是什么类型的陈述和提示技术，LLMs 与人类之间存在显著的陈述一致性。然而，LLMs 系统地高估了无意义陈述的深度，除了 Tk-instruct 外，它是唯一低估陈述深度的模型。只有少量训练样本的提示将 LLMs 的评分接近人类，而非思维链式的提示将 LLMs 的评分远离人类。此外，本研究揭示了强化学习来自人类反馈所引发的潜在偏见，即高估陈述的深度。