Large language models (LLMs) have significantly advanced the field of
artificial intelligence. Yet, evaluating them comprehensively remains
challenging. We argue that this is partly due to the predominant focus on
performance metrics in most benchmarks. This paper introduces CogBench, a
benchmark that includes ten behavioral metrics derived from seven cognitive
psychology experiments. This novel approach offers a toolkit for phenotyping
LLMs' behavior. We apply CogBench to 35 LLMs, yielding a rich and diverse
dataset. We analyze this data using statistical multilevel modeling techniques,
accounting for the nested dependencies among fine-tuned versions of specific
LLMs. Our study highlights the crucial role of model size and reinforcement
learning from human feedback (RLHF) in improving performance and aligning with
human behavior. Interestingly, we find that open-source models are less
risk-prone than proprietary models and that fine-tuning on code does not
necessarily enhance LLMs' behavior. Finally, we explore the effects of
prompt-engineering techniques. We discover that chain-of-thought prompting
improves probabilistic reasoning, while take-a-step-back prompting fosters
model-based behaviors.

大型语言模型在人工智能领域取得了显著进展，然而它们的综合评估仍具挑战性。本研究引入了 CogBench 评估工具，包含七项认知心理学实验中衍生出的十项行为度量指标，并运用这一工具对 35 个大型语言模型进行了分析。研究结果突出了模型规模和人类反馈的强化学习在性能提升和与人类行为的一致性方面的关键作用。此外，本研究还探讨了提示工程技术的影响，发现链式思路提示能够改善概率推理，而退一步思考提示则促进了基于模型的行为。

CogBench：一个大型语言模型进入心理学实验室

CogBench: a large language model walks into a psychology lab

Research has proven that stress reduces quality of life and causes many
diseases. For this reason, several researchers devised stress detection systems
based on physiological parameters. However, these systems require that
obtrusive sensors are continuously carried by the user. In our paper, we
propose an alternative approach providing evidence that daily stress can be
reliably recognized based on behavioral metrics, derived from the user's mobile
phone activity and from additional indicators, such as the weather conditions
(data pertaining to transitory properties of the environment) and the
personality traits (data concerning permanent dispositions of individuals). Our
multifactorial statistical model, which is person-independent, obtains the
accuracy score of 72.28% for a 2-class daily stress recognition problem. The
model is efficient to implement for most of multimedia applications due to
highly reduced low-dimensional feature space (32d). Moreover, we identify and
discuss the indicators which have strong predictive power.

通过分析用户的移动电话活动和额外指标，如天气条件和人格特征等数据，我们提出了一种替代方法，可靠地识别日常压力水平，并且高效易实现。