This study examines whether the attention scores between tokens in the BERT model significantly vary based on lexical categories during the fine-tuning process for downstream tasks. Drawing inspiration from the notion that in human language processing, syntactic and semantic information is parsed differently, we categorize tokens in sentences according to their lexical categories and focus on changes in attention scores among these categories. Our hypothesis posits that in downstream tasks that prioritize semantic information, attention scores centered on content words are enhanced, while in cases emphasizing syntactic information, attention scores centered on function words are intensified. Through experimentation conducted on six tasks from the GLUE benchmark dataset, we substantiate our hypothesis regarding the fine-tuning process. Furthermore, our additional investigations reveal the presence of BERT layers that consistently assign more bias to specific lexical categories, irrespective of the task, highlighting the existence of task-agnostic lexical category preferences.

本研究通过对BERT模型进行细调过程中，考察词汇类别对注意力分数的影响，进一步验证了重点语义信息的下游任务中注意力分数主要集中在内容词上的假设，并揭示了BERT层对特定词汇类别具有一致偏好的存在。

BERT模型中的注意力分数对GLUE基准上的句法和语义任务中的词类意识研究