Recent evaluations of LLMs on coreference resolution have revealed that traditional output formats and evaluation metrics do not fully capture the models' referential understanding. To address this, we introduce IdentifyMe, a new benchmark for mention resolution presented in a multiple-choice question (MCQ) format, commonly used for evaluating LLMs. IdentifyMe features long narratives and employs heuristics to exclude easily identifiable mentions, creating a more challenging task. The benchmark also consists of a curated mixture of different mention types and corresponding entities, allowing for a fine-grained analysis of model performance. We evaluate both closed- and open source LLMs on IdentifyMe and observe a significant performance gap (20-30%) between the state-of-the-art sub-10B open models vs. closed ones. We observe that pronominal mentions, which have limited surface information, are typically much harder for models to resolve than nominal mentions. Additionally, we find that LLMs often confuse entities when their mentions overlap in nested structures. The highest-scoring model, GPT-4o, achieves 81.9% accuracy, highlighting the strong referential capabilities of state-of-the-art LLMs while also indicating room for further improvement.

本研究针对传统输出格式和评估指标无法充分捕捉大规模语言模型（LLM）在共指解析中的参照理解这一问题，提出了新的基准IdentifyMe，该基准采用多项选择题形式并包含长篇叙述。通过提供不同类型的提及和相应的实体，我们细致评估了模型性能，并发现开放模型与封闭模型间存在显著绩效差距，尤其是在解析人称代词时表现尤为困难。

IdentifyMe：一项具有挑战性的长文本提及解析基准