Large language models (LLMs) are increasingly used to meet user information
needs, but their effectiveness in dealing with user queries that contain
various types of ambiguity remains unknown, ultimately risking user trust and
satisfaction. To this end, we introduce CLAMBER, a benchmark for evaluating
LLMs using a well-organized taxonomy. Building upon the taxonomy, we construct
~12K high-quality data to assess the strengths, weaknesses, and potential risks
of various off-the-shelf LLMs. Our findings indicate the limited practical
utility of current LLMs in identifying and clarifying ambiguous user queries,
even enhanced by chain-of-thought (CoT) and few-shot prompting. These
techniques may result in overconfidence in LLMs and yield only marginal
enhancements in identifying ambiguity. Furthermore, current LLMs fall short in
generating high-quality clarifying questions due to a lack of conflict
resolution and inaccurate utilization of inherent knowledge. In this paper,
CLAMBER presents a guidance and promotes further research on proactive and
trustworthy LLMs. Our dataset is available at
this https URL

通过引入 CLAMBER，本研究构建了一个评估大型语言模型（LLMs）的基准，发现当前 LLMs 在识别和澄清用户不确定查询方面的实用性有限，同时提出进一步研究如何提高 LLMs 的质量，解决冲突并增加可信度的指导。