The current evaluation of Large Language Models (LLMs) predominantly relies
on benchmarks focusing on their embedded knowledge by testing through
multiple-choice questions (MCQs), a format inherently suited for automated
evaluation. Our study extends this evaluation to explore LLMs' pragmatic
competence--a facet previously underexamined before the advent of sophisticated
LLMs, specifically in the context of Korean. We employ two distinct evaluation
setups: the conventional MCQ format, adapted for automatic evaluation, and
Open-Ended Questions (OEQs), assessed by human experts, to examine LLMs'
narrative response capabilities without predefined options. Our findings reveal
that GPT-4 excels, scoring 81.11 and 85.69 in the MCQ and OEQ setups,
respectively, with HyperCLOVA X, an LLM optimized for Korean, closely
following, especially in the OEQ setup, demonstrating a score of 81.56 with a
marginal difference of 4.13 points compared to GPT-4. Furthermore, while
few-shot learning strategies generally enhance LLM performance,
Chain-of-Thought (CoT) prompting introduces a bias toward literal
interpretations, hindering accurate pragmatic inference. Considering the
growing expectation for LLMs to understand and produce language that aligns
with human communicative norms, our findings emphasize the importance for
advancing LLMs' abilities to grasp and convey sophisticated meanings beyond
mere literal interpretations.

通过评估 Large Language Models（LLMs）的现状，我们探讨了 LLMs 在韩语背景下的语用能力，包括常规的多项选择题和开放式问题的评估，结果显示 GPT-4 在两种评估设置中表现出色，分别获得 81.11 和 85.69 分，HyperCLOVA X 也得到了良好的分数，尤其在开放式问题的评估中，获得了 81.56 分，与 GPT-4 相比仅相差 4.13 分，而使用 CoT 提示的少样本学习策略则引入了对字面解释的偏见，限制了准确的语用推理能力，这些发现强调了提升 LLMs 在理解和传达超出字面解释的复杂意义方面的重要性。