Language understanding is a multi-faceted cognitive capability, which the
Natural Language Processing (NLP) community has striven to model
computationally for decades. Traditionally, facets of linguistic intelligence
have been compartmentalized into tasks with specialized model architectures and
corresponding evaluation protocols. With the advent of large language models
(LLMs) the community has witnessed a dramatic shift towards general purpose,
task-agnostic approaches powered by generative models. As a consequence, the
traditional compartmentalized notion of language tasks is breaking down,
followed by an increasing challenge for evaluation and analysis. At the same
time, LLMs are being deployed in more real-world scenarios, including
previously unforeseen zero-shot setups, increasing the need for trustworthy and
reliable systems. Therefore, we argue that it is time to rethink what
constitutes tasks and model evaluation in NLP, and pursue a more holistic view
on language, placing trustworthiness at the center. Towards this goal, we
review existing compartmentalized approaches for understanding the origins of a
model's functional capacity, and provide recommendations for more multi-faceted
evaluation protocols.

语言理解是一个多方面的认知能力，自然语言处理（NLP）领域几十年来一直致力于计算化建模。最近大型语言模型（LLMs）的出现，使得以生成模型为动力的通用性、任务无关的方法成为主流，这导致了语言任务传统上的划分不再适用，带来了对评估和分析的挑战，同时也加大了对可靠系统的需求。因此，我们主张重新思考 NLP 中任务和模型评估的涵义，并追求对语言的更全面视角，将可靠性放在核心地位。为了实现这一目标，我们回顾了现有的划分式方法，以了解模型功能能力的起源，并提出了多方面评估协议的建议。