This paper proposes a transformer over transformer framework, called
Transformer$^2$, to perform neural text segmentation. It consists of two
components: bottom-level sentence encoders using pre-trained transformers, and
an upper-level transformer-based segmentation model based on the sentence
embeddings. The bottom-level component transfers the pre-trained knowledge
learned from large external corpora under both single and pair-wise supervised
NLP tasks to model the sentence embeddings for the documents. Given the
sentence embeddings, the upper-level transformer is trained to recover the
segmentation boundaries as well as the topic labels of each sentence. Equipped
with a multi-task loss and the pre-trained knowledge, Transformer$^2$ can
better capture the semantic coherence within the same segments. Our experiments
show that (1) Transformer$^2$ manages to surpass state-of-the-art text
segmentation models in terms of a commonly-used semantic coherence measure; (2)
in most cases, both single and pair-wise pre-trained knowledge contribute to
the model performance; (3) bottom-level sentence encoders pre-trained on
specific languages yield better performance than those pre-trained on specific
domains.

本文提出了一种名为 Transformer$^2$ 的基于 transformer 的 transformer 框架，用于进行神经文本分割，并在实验中证明了其在语义一致性方面的优越性。