This study analyzes changes in the attention mechanisms of large language
models (LLMs) when used to understand natural conversations between humans
(human-human). We analyze three use cases of LLMs: interactions over web
content, code, and mathematical texts. By analyzing attention distance,
dispersion, and interdependency across these domains, we highlight the unique
challenges posed by conversational data. Notably, conversations require nuanced
handling of long-term contextual relationships and exhibit higher complexity
through their attention patterns. Our findings reveal that while language
models exhibit domain-specific attention behaviors, there is a significant gap
in their ability to specialize in human conversations. Through detailed
attention entropy analysis and t-SNE visualizations, we demonstrate the need
for models trained with a diverse array of high-quality conversational data to
enhance understanding and generation of human-like dialogue. This research
highlights the importance of domain specialization in language models and
suggests pathways for future advancement in modeling human conversational
nuances.

本研究通过分析大型语言模型（LLMs）在理解人际对话时的注意机制变化，对与网络内容、代码和数学文本的三种使用情况进行了分析。研究表明，对话数据具有长期上下文关系的细致处理和通过注意力模式的高复杂性等独特挑战。我们的研究结果显示，虽然语言模型表现出领域特定的注意行为，但在专攻人类对话方面仍存在显著差距。通过详细的注意熵分析和 t-SNE 可视化，我们展示了训练于多样且高质量对话数据的模型，以提升对人类对话的理解和生成。这项研究强调了语言模型领域专长的重要性，并提出了未来建模人类对话细微差别的途径。