Text encoding is one of the most important steps in Natural Language
Processing (NLP). It has been done well by the self-attention mechanism in the
current state-of-the-art Transformer encoder, which has brought about
significant improvements in the performance of many NLP tasks. Though the
Transformer encoder may effectively capture general information in its
resulting representations, the backbone information, meaning the gist of the
input text, is not specifically focused on. In this paper, we propose explicit
and implicit text compression approaches to enhance the Transformer encoding
and evaluate models using this approach on several typical downstream tasks
that rely on the encoding heavily. Our explicit text compression approaches use
dedicated models to compress text, while our implicit text compression approach
simply adds an additional module to the main model to handle text compression.
We propose three ways of integration, namely backbone source-side fusion,
target-side fusion, and both-side fusion, to integrate the backbone information
into Transformer-based models for various downstream tasks. Our evaluation on
benchmark datasets shows that the proposed explicit and implicit text
compression approaches improve results in comparison to strong baselines. We
therefore conclude, when comparing the encodings to the baseline models, text
compression helps the encoders to learn better language representations.

本文提出了显式和隐式文本压缩方法来增强 Transformer 编码，并用这种方法评估了多个依赖于编码的典型下游任务的模型。我们提出了三种方式的集成，即骨干源侧融合，目标侧融合和双侧融合，以便于变压器基于模型集成骨干信息的各种下游任务中。与强基线相比较，我们的显式和隐式文本压缩方法改善了结果，我们认为文本压缩有助于编码器学习更好的语言表示。