We propose ConVEx (Conversational Value Extractor), an efficient pretraining
and fine-tuning neural approach for slot-labeling dialog tasks. Instead of
relying on more general pretraining objectives from prior work (e.g., language
modeling, response selection), ConVEx's pretraining objective, a novel pairwise
cloze task using Reddit data, is well aligned with its intended usage on
sequence labeling tasks. This enables learning domain-specific slot labelers by
simply fine-tuning decoding layers of the pretrained general-purpose sequence
labeling model, while the majority of the pretrained model's parameters are
kept frozen. We report state-of-the-art performance of ConVEx across a range of
diverse domains and data sets for dialog slot-labeling, with the largest gains
in the most challenging, few-shot setups. We believe that ConVEx's reduced
pretraining times (i.e., only 18 hours on 12 GPUs) and cost, along with its
efficient fine-tuning and strong performance, promise wider portability and
scalability for data-efficient sequence-labeling tasks in general.

我们提出了 ConVEx（面向对话任务的槽标记对话值提取器），通过 Reddit 数据上的新颖成对填空任务的预训练目标，实现序列标记任务的领域特定槽标记器的学习，并在保持预训练模型的大多数参数固定不变的情况下，仅通过微调预训练的通用序列标记模型的解码层来实现。