Transformer-based approaches have been successfully used to obtain
state-of-the-art accuracy on natural language processing (NLP) tasks with
semi-structured tables. These model architectures are typically deep, resulting
in slow training and inference, especially for long inputs. To improve
efficiency while maintaining a high accuracy, we propose a new architecture,
DoT, a double transformer model, that decomposes the problem into two
sub-tasks: A shallow pruning transformer that selects the top-K tokens,
followed by a deep task-specific transformer that takes as input those K
tokens. Additionally, we modify the task-specific attention to incorporate the
pruning scores. The two transformers are jointly trained by optimizing the
task-specific loss. We run experiments on three benchmarks, including
entailment and question-answering. We show that for a small drop of accuracy,
DoT improves training and inference time by at least 50%. We also show that the
pruning transformer effectively selects relevant tokens enabling the end-to-end
model to maintain similar accuracy as slower baseline models. Finally, we
analyse the pruning and give some insight into its impact on the task model.

本研究提出了一种名为 DoT 的双变形器模型，其中一个较浅的修剪变形器用于选择最佳 K 个关键词，然后一个较深的任务特定变形器将这 K 个关键词作为输入，通过优化任务特定的损失函数，两个变形器共同训练。 DoT 模型虽然产生了轻微的精度降低，但提高了训练和推理时间至少 50％。