In this study, we investigate using graph neural network (GNN)
representations to enhance contextualized representations of pre-trained
language models (PLMs) for keyphrase extraction from lengthy documents. We show
that augmenting a PLM with graph embeddings provides a more comprehensive
semantic understanding of words in a document, particularly for long documents.
We construct a co-occurrence graph of the text and embed it using a graph
convolutional network (GCN) trained on the task of edge prediction. We propose
a graph-enhanced sequence tagging architecture that augments contextualized PLM
embeddings with graph representations. Evaluating on benchmark datasets, we
demonstrate that enhancing PLMs with graph embeddings outperforms
state-of-the-art models on long documents, showing significant improvements in
F1 scores across all the datasets. Our study highlights the potential of GNN
representations as a complementary approach to improve PLM performance for
keyphrase extraction from long documents.

本研究探讨使用图神经网络 (GNN) 表示来增强预训练语言模型 (PLMs) 的上下文表示，用于从长文档中提取关键词。我们构建了文本的共现图，并使用在边预测任务上训练的图卷积网络 (GCN) 进行嵌入。我们提出了一种增强序列标记体系结构来增强 PLM 嵌入与图嵌入。在基准数据集上评估，我们证明了使用图嵌入增强 PLM 在长文档上的性能优于现有最先进模型，并在所有数据集上显示了 F1 分数的显著改进。我们的研究突显了 GNN 表示的潜力，作为改善从长文档中提取关键词时 PLM 性能的补充方法。

使用图嵌入增强从长科学文档中提取关键词

Enhancing Keyphrase Extraction from Long Scientific Documents using  Graph Embeddings

Learned vector representations of words are useful tools for many information
retrieval and natural language processing tasks due to their ability to capture
lexical semantics. However, while many such tasks involve or even rely on named
entities as central components, popular word embedding models have so far
failed to include entities as first-class citizens. While it seems intuitive
that annotating named entities in the training corpus should result in more
intelligent word features for downstream tasks, performance issues arise when
popular embedding approaches are naively applied to entity annotated corpora.
Not only are the resulting entity embeddings less useful than expected, but one
also finds that the performance of the non-entity word embeddings degrades in
comparison to those trained on the raw, unannotated corpus. In this paper, we
investigate approaches to jointly train word and entity embeddings on a large
corpus with automatically annotated and linked entities. We discuss two
distinct approaches to the generation of such embeddings, namely the training
of state-of-the-art embeddings on raw-text and annotated versions of the
corpus, as well as node embeddings of a co-occurrence graph representation of
the annotated corpus. We compare the performance of annotated embeddings and
classical word embeddings on a variety of word similarity, analogy, and
clustering evaluation tasks, and investigate their performance in
entity-specific tasks. Our findings show that it takes more than training
popular word embedding models on an annotated corpus to create entity
embeddings with acceptable performance on common test cases. Based on these
results, we discuss how and when node embeddings of the co-occurrence graph
representation of the text can restore the performance.

本文研究了在大型语料库上联合训练单词和实体嵌入的方法以及它们在各种自然语言处理任务上的表现，发现仅仅在注释语料库上训练流行的单词嵌入模型无法创建具有可接受性能的实体嵌入，而我们的实体共现图嵌入方法能够恢复性能。