In real-world scenarios, labeled samples for dialogue summarization are
usually limited (i.e., few-shot) due to high annotation costs for high-quality
dialogue summaries. To efficiently learn from few-shot samples, previous works
have utilized massive annotated data from other downstream tasks and then
performed prompt transfer in prompt tuning so as to enable cross-task knowledge
transfer. However, existing general-purpose prompt transfer techniques lack
consideration for dialogue-specific information. In this paper, we focus on
improving the prompt transfer from dialogue state tracking to dialogue
summarization and propose Skeleton-Assisted Prompt Transfer (SAPT), which
leverages skeleton generation as extra supervision that functions as a medium
connecting the distinct source and target task and resulting in the model's
better consumption of dialogue state information. To automatically extract
dialogue skeletons as supervised training data for skeleton generation, we
design a novel approach with perturbation-based probes requiring neither
annotation effort nor domain knowledge. Training the model on such skeletons
can also help preserve model capability during prompt transfer. Our method
significantly outperforms existing baselines. In-depth analyses demonstrate the
effectiveness of our method in facilitating cross-task knowledge transfer in
few-shot dialogue summarization.

本研究提出了一种 Skeleton-Assisted Prompt Transfer（SAPT）方法，利用生成骨架作为额外的监督信息来实现对话状态信息的更好使用，以促进少样本对话摘要中的跨任务知识转移。通过基于扰动的探针的新方法自动提取对话框架作为有监督的训练数据，避免了注释工作和领域知识的需求。研究表明，我们的方法显著优于现有的基准，并在促进跨任务知识转移中发挥了作用。

通过骨架辅助提示传递进行少样本对话摘要

Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer

This technical report briefly describes our JDExplore d-team's Vega v2
submission on the SuperGLUE leaderboard. SuperGLUE is more challenging than the
widely used general language understanding evaluation (GLUE) benchmark,
containing eight difficult language understanding tasks, including question
answering, natural language inference, word sense disambiguation, coreference
resolution, and reasoning. [Method] Instead of arbitrarily increasing the size
of a pretrained language model (PLM), our aim is to 1) fully extract knowledge
from the input pretraining data given a certain parameter budget, e.g., 6B, and
2) effectively transfer this knowledge to downstream tasks. To achieve goal 1),
we propose self-evolution learning for PLMs to wisely predict the informative
tokens that should be masked, and supervise the masked language modeling (MLM)
process with rectified smooth labels. For goal 2), we leverage the prompt
transfer technique to improve the low-resource tasks by transferring the
knowledge from the foundation model and related downstream tasks to the target
task. [Results] According to our submission record (Oct. 2022), with our
optimized pretraining and fine-tuning strategies, our 6B Vega method achieved
new state-of-the-art performance on 4/8 tasks, sitting atop the SuperGLUE
leaderboard on Oct. 8, 2022, with an average score of 91.3.

该研究报告描述了一种基于自进化学习的预训练语言模型和 prompt transfer 技术，将知识转移到下游任务以实现超越 SuperGLUE 排行榜的表现。2022 年 10 月，该方法在 4/8 个任务上实现了最新的最先进性能，平均得分为 91.3。

自进化：实现语言模型高效预训练与下游适应的一项探究

Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE

Prompt-tuning, which freezes pretrained language models (PLMs) and only
fine-tunes few parameters of additional soft prompt, shows competitive
performance against full-parameter fine-tuning (i.e.model-tuning) when the PLM
has billions of parameters, but still performs poorly in the case of smaller
PLMs. Hence, prompt transfer (PoT), which initializes the target prompt with
the trained prompt of similar source tasks, is recently proposed to improve
over prompt-tuning. However, such a vanilla PoT approach usually achieves
sub-optimal performance, as (i) the PoT is sensitive to the similarity of
source-target pair and (ii) directly fine-tuning the prompt initialized with
source prompt on target task might lead to catastrophic forgetting of source
knowledge. In response to these problems, we propose a new metric to accurately
predict the prompt transferability (regarding (i)), and a novel PoT approach
(namely PANDA) that leverages the knowledge distillation technique to transfer
the "knowledge" from the source prompt to the target prompt in a subtle manner
and alleviate the catastrophic forgetting effectively (regarding (ii)).
Furthermore, to achieve adaptive prompt transfer for each source-target pair,
we use our metric to control the knowledge transfer in our PANDA approach.
Extensive and systematic experiments on 189 combinations of 21 source and 9
target datasets across 5 scales of PLMs demonstrate that: 1) our proposed
metric works well to predict the prompt transferability; 2) our PANDA
consistently outperforms the vanilla PoT approach by 2.3% average score (up to
24.1%) among all tasks and model sizes; 3) with our PANDA approach,
prompt-tuning can achieve competitive and even better performance than
model-tuning in various PLM scales scenarios. Code and models will be released
upon acceptance.

该研究提出了一种新的度量方法和 PANDA 方法来解决预训练语言模型中的 prompt 转移问题，PANDA 方法使用知识蒸馏技术来迁移源 prompt 中的知识到目标 prompt 中，从而避免源知识的灾难性遗忘。实验证明，该方法在各种 PLM 尺度场景下的性能均优于传统的 prompt 转移方法和模型调参方法。