Autoregressive models used to generate responses in open-domain dialogue
systems often struggle to take long-term context into account and to maintain
consistency over a dialogue. Previous research in open-domain dialogue
generation has shown that the use of \emph{auxiliary tasks} can introduce
inductive biases that encourage the model to improve these qualities. However,
most previous research has focused on encoder-only or encoder/decoder models,
while the use of auxiliary tasks in \emph{decoder-only} autoregressive models
is under-explored. This paper describes an investigation where four different
auxiliary tasks are added to small and medium-sized GPT-2 models fine-tuned on
the PersonaChat and DailyDialog datasets. The results show that the
introduction of the new auxiliary tasks leads to small but consistent
improvement in evaluations of the investigated models.

研究使用辅助任务对自回归模型进行改进，提高其在开放域对话生成中的长期语境处理和一致性维护，结果在小型和中型的 GPT-2 模型上表明引入新的辅助任务虽然有所改进，但仍有发展空间。

多任务学习在开放域对话系统中的实证研究

An Empirical Study of Multitask Learning to Improve Open Domain Dialogue  Systems

We present a novel approach to generating news headlines in Finnish for a
given news story. We model this as a summarization task where a model is given
a news article, and its task is to produce a concise headline describing the
main topic of the article. Because there are no openly available GPT-2 models
for Finnish, we will first build such a model using several corpora. The model
is then fine-tuned for the headline generation task using a massive news
corpus. The system is evaluated by 3 expert journalists working in a Finnish
media house. The results showcase the usability of the presented approach as a
headline suggestion tool to facilitate the news production process.

我们提出了一种新颖的方法，以生成针对给定新闻事件的芬兰语新闻标题为主，通过构建一个模型，该模型使用多个语料库训练，在一个巨大的新闻语料库的帮助下，对其进行微调以进行标题生成任务，并由 3 名芬兰媒体机构的专业记者进行评估，结果表明所提出的方法在作为新闻生产过程中的标题建议工具方面具有可用性。

芬兰语事实新闻标题的自动生成

Automatic Generation of Factual News Headlines in Finnish

Pre-trained language models have shown remarkable success in improving
various downstream NLP tasks due to their ability to capture dependencies in
textual data and generate natural responses. In this paper, we leverage the
power of pre-trained language models for improving video-grounded dialogue,
which is very challenging and involves complex features of different dynamics:
(1) Video features which can extend across both spatial and temporal
dimensions; and (2) Dialogue features which involve semantic dependencies over
multiple dialogue turns. We propose a framework by extending GPT-2 models to
tackle these challenges by formulating video-grounded dialogue tasks as a
sequence-to-sequence task, combining both visual and textual representation
into a structured sequence, and fine-tuning a large pre-trained GPT-2 network.
Our framework allows fine-tuning language models to capture dependencies across
multiple modalities over different levels of information: spatio-temporal level
in video and token-sentence level in dialogue context. We achieve promising
improvement on the Audio-Visual Scene-Aware Dialogues (AVSD) benchmark from
DSTC7, which supports a potential direction in this line of research.

本文提出了一种基于 GPT-2 模型的框架，将视频与文本表示结合成连续、有结构的序列，并利用其 fine-tuning 能力来解决视频对话中的挑战，从而在 Audio-Visual Scene-Aware Dialogues 基准测试中取得了显著的改进。