A desirable dialog system should be able to continually learn new skills
without forgetting old ones, and thereby adapt to new domains or tasks in its
life cycle. However, continually training a model often leads to a well-known
catastrophic forgetting issue. In this paper, we present Continual Prompt
Tuning, a parameter-efficient framework that not only avoids forgetting but
also enables knowledge transfer between tasks. To avoid forgetting, we only
learn and store a few prompt tokens' embeddings for each task while freezing
the backbone pre-trained model. To achieve bi-directional knowledge transfer
among tasks, we propose several techniques (continual prompt initialization,
query fusion, and memory replay) to transfer knowledge from preceding tasks and
a memory-guided technique to transfer knowledge from subsequent tasks.
Extensive experiments demonstrate the effectiveness and efficiency of our
proposed method on continual learning for dialog state tracking, compared with
state-of-the-art baselines.

本文提出了一种连续的提示调整参数有效的框架，可避免对旧任务的遗忘并在任务之间实现知识转移，验证实验表明该方法在对话状态跟踪领域中具有有效性和高效性。

对话状态跟踪的继续提示调整

Continual Prompt Tuning for Dialog State Tracking

More transformer blocks with residual connections have recently achieved
impressive results on various tasks. To achieve better performance with fewer
trainable parameters, recent methods are proposed to go shallower by parameter
sharing or model compressing along with the depth. However, weak modeling
capacity limits their performance. Contrastively, going wider by inducing more
trainable matrixes and parameters would produce a huge model requiring advanced
parallelism to train and inference.
In this paper, we propose a parameter-efficient framework, going wider
instead of deeper. Specially, following existing works, we adapt parameter
sharing to compress along depth. But, such deployment would limit the
performance. To maximize modeling capacity, we scale along model width by
replacing feed-forward network (FFN) with mixture-of-experts (MoE). Across
transformer blocks, instead of sharing normalization layers, we propose to use
individual layernorms to transform various semantic representations in a more
parameter-efficient way. To evaluate our plug-and-run framework, we design
WideNet and conduct comprehensive experiments on popular computer vision and
natural language processing benchmarks. On ImageNet-1K, our best model
outperforms Vision Transformer (ViT) by $1.5\%$ with $0.72 \times$ trainable
parameters. Using $0.46 \times$ and $0.13 \times$ parameters, our WideNet can
still surpass ViT and ViT-MoE by $0.8\%$ and $2.1\%$, respectively. On four
natural language processing datasets, WideNet outperforms ALBERT by $1.8\%$ on
average and surpass BERT using factorized embedding parameterization by $0.8\%$
with fewer parameters.

本文提出了一种宽架构、混合专家 (MoE) 替代前馈神经网络 (FFN) 的参数高效框架，通过参数共享以压缩深度，并使用独立的 layernorms 来转换各种语义表示，实验结果在多个计算机视觉和自然语言处理基准测试中表现优异，最佳模型能以 0.72 倍的可训练参数超越 Vision Transformer 1.5％、超越 ALBERT 1.8％平均表现和使用分解嵌入参数化的 BERT 0.8％。