This paper addresses the problem of incremental domain adaptation (IDA) in
natural language processing (NLP). We assume each domain comes one after
another, and that we could only access data in the current domain. The goal of
IDA is to build a unified model performing well on all the domains that we have
encountered. We adopt the recurrent neural network (RNN) widely used in NLP,
but augment it with a directly parameterized memory bank, which is retrieved by
an attention mechanism at each step of RNN transition. The memory bank provides
a natural way of IDA: when adapting our model to a new domain, we progressively
add new slots to the memory bank, which increases the number of parameters, and
thus the model capacity. We learn the new memory slots and fine-tune existing
parameters by back-propagation. Experimental results show that our approach
achieves significantly better performance than fine-tuning alone. Compared with
expanding hidden states, our approach is more robust for old domains, shown by
both empirical and theoretical results. Our model also outperforms previous
work of IDA including elastic weight consolidation and progressive neural
networks in the experiments.

本文提出了一种增量域自适应的方法，它采用了在自然语言处理中广泛使用的递归神经网络，并将它与参数化的内存库结合起来，通过每次 RNN 转移步骤进行检索和更新，以构建一个在所有域上表现良好的统一模型。实验结果表明，与 fine-tuning 单独方法相比，我们的方法表现明显更好。