Various works have been extensively studied in the research of text-to-image
generation. Although existing models perform well in text-to-image generation,
there are significant challenges when directly employing them to generate
images in dialogs. In this paper, we first highlight a new problem:
dialog-to-image generation, that is, given the dialog context, the model should
generate a realistic image which is consistent with the specified conversation
as response. To tackle the problem, we propose an efficient approach for
dialog-to-image generation without any intermediate translation, which
maximizes the extraction of the semantic information contained in the dialog.
Considering the characteristics of dialog structure, we put segment token
before each sentence in a turn of a dialog to differentiate different speakers.
Then, we fine-tune pre-trained text-to-image models to enable them to generate
images conditioning on processed dialog context. After fine-tuning, our
approach can consistently improve the performance of various models across
multiple metrics. Experimental results on public benchmark demonstrate the
effectiveness and practicability of our method.

通过语料上进行的实验结果证明，我们的方法有效且实用，能够生成与指定对话一致的逼真图像，并在多个度量指标上持续提高各种模型的性能。

教授文本到图像模型进行交流

Teaching Text-to-Image Models to Communicate

Existing speech recognition systems are typically built at the sentence
level, although it is known that dialog context, e.g. higher-level knowledge
that spans across sentences or speakers, can help the processing of long
conversations. The recent progress in end-to-end speech recognition systems
promises to integrate all available information (e.g. acoustic, language
resources) into a single model, which is then jointly optimized. It seems
natural that such dialog context information should thus also be integrated
into the end-to-end models to improve further recognition accuracy. In this
work, we present a dialog-context aware speech recognition model, which
explicitly uses context information beyond sentence-level information, in an
end-to-end fashion. Our dialog-context model captures a history of
sentence-level context so that the whole system can be trained with
dialog-context information in an end-to-end manner. We evaluate our proposed
approach on the Switchboard conversational speech corpus and show that our
system outperforms a comparable sentence-level end-to-end speech recognition
system.

本研究提出了一种对话上下文感知的语音识别模型，该模型可以在端到端的方式下明确地使用除句子级别信息以外的上下文信息，以使整个系统能够被训练得更好并表现得更优秀。我们在 Switchboard 交谈语音语料库上评估了我们的方法并显示出优于可比较的句子级端到端语音识别系统的效果。