Multilingual neural machine translation with a single model has drawn much
attention due to its capability to deal with multiple languages. However, the
current multilingual translation paradigm often makes the model tend to
preserve the general knowledge, but ignore the language-specific knowledge.
Some previous works try to solve this problem by adding various kinds of
language-specific modules to the model, but they suffer from the parameter
explosion problem and require specialized manual design. To solve these
problems, we propose to divide the model neurons into general and
language-specific parts based on their importance across languages. The general
part is responsible for preserving the general knowledge and participating in
the translation of all the languages, while the language-specific part is
responsible for preserving the language-specific knowledge and participating in
the translation of some specific languages. Experimental results on several
language pairs, covering IWSLT and Europarl corpus datasets, demonstrate the
effectiveness and universality of the proposed method.

本篇研究论文提出了一种新的多语言神经机器翻译模型，将模型神经元划分为通用部分和特定语言部分，以解决当前的多语言翻译方法忽略语言特定知识的问题。实验证明，该方法对于多种语言对有很好的效果和普适性。

基于重要性的神经元分配在多语言神经机器翻译中的应用

Importance-based Neuron Allocation for Multilingual Neural Machine  Translation

Neural dialogue models suffer from low-quality responses when interacted in
practice, demonstrating difficulty in generalization beyond training data.
Recently, knowledge distillation has been used to successfully regularize the
student by transferring knowledge from the teacher. However, the teacher and
the student are trained on the same dataset and tend to learn similar feature
representations, whereas the most general knowledge should be found through
differences. The finding of general knowledge is further hindered by the
unidirectional distillation, as the student should obey the teacher and may
discard some knowledge that is truly general but refuted by the teacher. To
this end, we propose a novel training framework, where the learning of general
knowledge is more in line with the idea of reaching consensus, i.e., finding
common knowledge that is beneficial to different yet all datasets through
diversified learning partners. Concretely, the training task is divided into a
group of subtasks with the same number of students. Each student assigned to
one subtask not only is optimized on the allocated subtask but also imitates
multi-view feature representation aggregated from other students (i.e., student
peers), which induces students to capture common knowledge among different
subtasks and alleviates the over-fitting of students on the allocated subtasks.
To further enhance generalization, we extend the unidirectional distillation to
the bidirectional distillation that encourages the student and its student
peers to co-evolve by exchanging complementary knowledge with each other.
Empirical results and analysis demonstrate that our training framework
effectively improves the model generalization without sacrificing training
efficiency.

本文提出一种基于知识蒸馏的对话模型训练框架，通过分组多任务学习、多视角特征蒸馏和双向蒸馏等策略，找到多数据集共性知识以提高模型泛化性能。实验结果表明该框架可以有效地提高模型的泛化性能，而不影响训练效率。

双向蒸馏的多视角特征表示在对话生成中的应用

Multi-View Feature Representation for Dialogue Generation with  Bidirectional Distillation

To bridge the gap between Machine Reading Comprehension (MRC) models and
human beings, which is mainly reflected in the hunger for data and the
robustness to noise, in this paper, we explore how to integrate the neural
networks of MRC models with the general knowledge of human beings. On the one
hand, we propose a data enrichment method, which uses WordNet to extract
inter-word semantic connections as general knowledge from each given
passage-question pair. On the other hand, we propose an end-to-end MRC model
named as Knowledge Aided Reader (KAR), which explicitly uses the above
extracted general knowledge to assist its attention mechanisms. Based on the
data enrichment method, KAR is comparable in performance with the
state-of-the-art MRC models, and significantly more robust to noise than them.
When only a subset (20%-80%) of the training examples are available, KAR
outperforms the state-of-the-art MRC models by a large margin, and is still
reasonably robust to noise.

本文提出了一种基于数据丰富的方法，提取给定语料对中的语义联系作为人类的通用知识，并在此基础上构建了一种知识辅助的 MRC 模型，与业内领先 MRC 模型相比具有相当的性能和更强的抗干扰能力。