The boardgame Diplomacy is a challenging setting for communicative and
cooperative artificial intelligence. The most prominent communicative Diplomacy
AI, Cicero, has excellent strategic abilities, exceeding human players.
However, the best Diplomacy players master communication, not just tactics,
which is why the game has received attention as an AI challenge. This work
seeks to understand the degree to which Cicero succeeds at communication.
First, we annotate in-game communication with abstract meaning representation
to separate in-game tactics from general language. Second, we run two dozen
games with humans and Cicero, totaling over 200 human-player hours of
competition. While AI can consistently outplay human players, AI-Human
communication is still limited because of AI's difficulty with deception and
persuasion. This shows that Cicero relies on strategy and has not yet reached
the full promise of communicative and cooperative AI.

通过将游戏内的交流与抽象意义表达进行注释，以及与人类进行二十四场比赛，总计超过 200 个小时的竞争，本研究旨在了解 Cicero 在交流上的成功程度，结果显示 Cicero 在策略上表现出色，但在隐瞒和说服方面仍存在困难，尚未实现完全的交流合作人工智能的潜力。

更多胜利，较少合作：评估西塞罗的外交策略

More Victories, Less Cooperation: Assessing Cicero's Diplomacy Play

Large language models (LLMs) have played a pivotal role in building
communicative AI to imitate human behaviors but face the challenge of efficient
customization. To tackle this challenge, recent studies have delved into the
realm of model editing, which manipulates specific memories of language models
and changes the related language generation. However, the robustness of model
editing remains an open question. This work seeks to understand the strengths
and limitations of editing methods, thus facilitating robust, realistic
applications of communicative AI. Concretely, we conduct extensive analysis to
address the three key research questions. Q1: Can edited LLMs behave
consistently resembling communicative AI in realistic situations? Q2: To what
extent does the rephrasing of prompts lead LLMs to deviate from the edited
knowledge memory? Q3: Which knowledge features are correlated with the
performance and robustness of editing? Our experimental results uncover a
substantial disparity between existing editing methods and the practical
application of LLMs. On rephrased prompts that are complex and flexible but
common in realistic applications, the performance of editing experiences a
significant decline. Further analysis shows that more popular knowledge is
memorized better, easier to recall, and more challenging to edit effectively.

大语言模型的编辑方法在实际应用中存在明显的性能衰退，更流行的知识难以编辑且对性能和鲁棒性有较强关联。