This thesis investigates how natural language understanding and generation
with transformer models can benefit from grounding the models with knowledge
representations and addresses the following key research questions: (i) Can
knowledge of entities extend its benefits beyond entity-centric tasks, such as
entity linking? (ii) How can we faithfully and effectively extract such
structured knowledge from raw text, especially noisy web text? (iii) How do
other types of knowledge, beyond structured knowledge, contribute to improving
NLP tasks?
Studies in this thesis find that incorporating relevant and up-to-date
knowledge of entities benefits fake news detection, and entity-focused
code-switching significantly enhances zero-shot cross-lingual transfer on
entity-centric tasks. In terms of effective and faithful approaches to
extracting structured knowledge, it is observed that integrating negative
examples and training with entity planning significantly improves performance.
Additionally, it is established that other general forms of knowledge, such as
parametric and distilled knowledge, enhance multimodal and multilingual
knowledge-intensive tasks. This research shows the tangible benefits of diverse
knowledge integration and motivates further exploration in this direction.

通过将转换器模型与知识表示相关联，本论文研究了自然语言理解和生成的效果，并回答了以下关键研究问题：(i) 实体知识是否能够超越实体链接等实体为中心的任务，并带来更多的好处？(ii) 如何从原始文本中准确有效地提取这种结构化知识，尤其是嘈杂的网络文本？(iii) 除了结构化知识，其他类型的知识如何对改进自然语言处理任务有所贡献？研究表明，引入相关和最新的实体知识有助于虚假新闻检测，而面向实体的语言转换显著提高了零样本跨语言迁移的效果。在提取结构化知识方面，将负面样本整合和与实体规划一起训练显著提高了性能。此外，研究还发现，其他一般形式的知识，如参数化知识和精炼知识，有助于多模态和多语言知识密集型任务。这项研究展示了多元知识整合的实际好处，并鼓励在这个方向上进一步探索。

走向知识驱动的自然语言理解与生成

Towards Knowledge-Grounded Natural Language Understanding and Generation

In recent years, Pre-trained Language Models (PLMs) have shown their
superiority by pre-training on unstructured text corpus and then fine-tuning on
downstream tasks. On entity-rich textual resources like Wikipedia,
Knowledge-Enhanced PLMs (KEPLMs) incorporate the interactions between tokens
and mentioned entities in pre-training, and are thus more effective on
entity-centric tasks such as entity linking and relation classification.
Although exploiting Wikipedia's rich structures to some extent, conventional
KEPLMs still neglect a unique layout of the corpus where each Wikipedia page is
around a topic entity (identified by the page URL and shown in the page title).
In this paper, we demonstrate that KEPLMs without incorporating the topic
entities will lead to insufficient entity interaction and biased (relation)
word semantics. We thus propose KEPLET, a novel Knowledge-Enhanced Pre-trained
LanguagE model with Topic entity awareness. In an end-to-end manner, KEPLET
identifies where to add the topic entity's information in a Wikipedia sentence,
fuses such information into token and mentioned entities representations, and
supervises the network learning, through which it takes topic entities back
into consideration. Experiments demonstrated the generality and superiority of
KEPLET which was applied to two representative KEPLMs, achieving significant
improvements on four entity-centric tasks.

本文提出了一种新型的 KEPLMs - KEPLET，它不仅考虑了维基百科丰富的语料库结构，还具有主题实体感知。实验证明，KEPLET 应用于两种典型的 KEPLMs，在四项实体中心任务中取得了显著的改进。