Previous works on Large Language Models (LLMs) have mainly focused on
evaluating their helpfulness or harmlessness. However, honesty, another crucial
alignment criterion, has received relatively less attention. Dishonest
behaviors in LLMs, such as spreading misinformation and defrauding users,
eroding user trust, and causing real-world harm, present severe risks that
intensify as these models approach superintelligence levels. Enhancing honesty
in LLMs addresses critical deficiencies and helps uncover latent capabilities
that are not readily expressed. This underscores the urgent need for reliable
methods and benchmarks to effectively ensure and evaluate the honesty of LLMs.
In this paper, we introduce BeHonest, a pioneering benchmark specifically
designed to assess honesty in LLMs comprehensively. BeHonest evaluates three
essential aspects of honesty: awareness of knowledge boundaries, avoidance of
deceit, and consistency in responses. Building on this foundation, we designed
10 scenarios to evaluate and analyze 9 popular LLMs on the market, including
both closed-source and open-source models from different model families with
varied model sizes. Our findings indicate that there is still significant room
for improvement in the honesty of LLMs. We also encourage the AI community to
prioritize honesty alignment in LLMs. Our benchmark and code can be found at:
https://github.com/GAIR-NLP/BeHonest.

这篇论文介绍了一个名为 BeHonest 的新基准，旨在全面评估大型语言模型（LLMs）的诚实性，并强调了 LLMs 诚实性对现实世界的影响和迫切需要可靠方法和基准来确保和评估 LLMs 的诚实性。