Natural language understanding includes the tasks of intent detection
(identifying a user's objectives) and slot filling (extracting the entities
relevant to those objectives). Prior slot filling methods assume that each
intent type cannot occur more than once within a message, however this is often
not a valid assumption for real-world settings. In this work, we generalize
slot filling by removing the constraint of unique intents in a message. We cast
this as a JSON generation task and approach it using a language model. We
create a pre-training dataset by combining DBpedia and existing slot filling
datasets that we convert for JSON generation. We also generate an in-domain
dataset using GPT-3. We train T5 models for this task (with and without
exemplars in the prompt) and find that both training datasets improve
performance, and that the model is able to generalize to intent types not seen
during training.

本文提出了一种更普适的槽位填充方法，通过将其视为 JSON 生成任务，并使用语言模型进行处理。本文结合 DBpedia 和现有的槽位填充数据集以及 GPT-3 生成属于该领域的数据集，使用 T5 模型进行训练，并发现加入 prompt 后两个训练集的性能都有所提高。最终模型可以泛化到训练期间未见过的意图类型。

通用多意图条件槽位填充

Generalized Multiple Intent Conditioned Slot Filling

Large language models (LLMs) have demonstrated human-level performance on a
vast spectrum of natural language tasks. However, it is largely unexplored
whether they can better internalize knowledge from a structured data, such as a
knowledge graph, or from text. In this work, we propose a method to infuse
structured knowledge into LLMs, by directly training T5 models on factual
triples of knowledge graphs (KGs). We show that models pre-trained on Wikidata
KG with our method outperform the T5 baselines on FreebaseQA and WikiHop, as
well as the Wikidata-answerable subset of TriviaQA and NaturalQuestions. The
models pre-trained on factual triples compare competitively with the ones on
natural language sentences that contain the same knowledge. Trained on a
smaller size KG, WikiMovies, we saw 3x improvement of exact match score on
MetaQA task compared to T5 baseline. The proposed method has an advantage that
no alignment between the knowledge graph and text corpus is required in
curating training data. This makes our method particularly useful when working
with industry-scale knowledge graphs.

本研究旨在探讨结构化数据如知识图谱和文本等对大语言模型（LLMs）内部化知识的差异，提出了一种将结构化知识注入 LLMs 的方法，并通过在不同知识图谱数据上的预训练和多个自然语言任务的性能比较，证明了该方法的有效性。

SKILL: 大语言模型结构化知识注入

SKILL: Structured Knowledge Infusion for Large Language Models

Data-to-text generation focuses on generating fluent natural language
responses from structured meaning representations (MRs). Such representations
are compositional and it is costly to collect responses for all possible
combinations of atomic meaning schemata, thereby necessitating few-shot
generalization to novel MRs. In this work, we systematically study the
compositional generalization of the state-of-the-art T5 models in few-shot
data-to-text tasks. We show that T5 models fail to generalize to unseen MRs,
and we propose a template-based input representation that considerably improves
the model's generalization capability. To further improve the model's
performance, we propose an approach based on self-training using fine-tuned
BLEURT for pseudo response selection. On the commonly-used SGD and Weather
benchmarks, the proposed self-training approach improves tree accuracy by 46%+
and reduces the slot error rates by 73%+ over the strong T5 baselines in
few-shot settings.

本文研究了结构化意义表示到自然语言生成中 T5 模型的泛化问题，提出了基于模板的输入表示和基于 fine-tuned BLEURT 的自训练方法来改善模型的泛化和性能，在少样本场景下通过常用的 SGD 和 Weather 基准测试取得了比强基线模型 46% 以上的提升和 73% 以上的错误率降低。