Transformer-based models have achieved great success on sentence pair
modeling tasks, such as answer selection and natural language inference (NLI).
These models generally perform cross-attention over input pairs, leading to
prohibitive computational costs. Recent studies propose dual-encoder and late
interaction architectures for faster computation. However, the balance between
the expressive of cross-attention and computation speedup still needs better
coordinated. To this end, this paper introduces a novel paradigm MixEncoder for
efficient sentence pair modeling. MixEncoder involves a light-weight
cross-attention mechanism. It conducts query encoding only once while modeling
the query-candidate interaction in parallel. Extensive experiments conducted on
four tasks demonstrate that our MixEncoder can speed up sentence pairing by
over 113x while achieving comparable performance as the more expensive
cross-attention models.

本文提出了一种高效的 MixEncoder 范式，它包括一种轻量级的交叉关注机制，可以在快速计算的同时实现可比较的性能，大大加速了句子配对。

一次就够：一种轻量级的交叉注意力用于快速语句对建模

Once is Enough: A Light-Weight Cross-Attention for Fast Sentence Pair Modeling

In this paper, we analyze several neural network designs (and their
variations) for sentence pair modeling and compare their performance
extensively across eight datasets, including paraphrase identification,
semantic textual similarity, natural language inference, and question answering
tasks. Although most of these models have claimed state-of-the-art performance,
the original papers often reported on only one or two selected datasets. We
provide a systematic study and show that (i) encoding contextual information by
LSTM and inter-sentence interactions are critical, (ii) Tree-LSTM does not help
as much as previously claimed but surprisingly improves performance on Twitter
datasets, (iii) the Enhanced Sequential Inference Model is the best so far for
larger datasets, while the Pairwise Word Interaction Model achieves the best
performance when less data is available. We release our implementations as an
open-source toolkit.

本文分析了几种神经网络设计（及其变体），对八个数据集进行了广泛的比较，包括释义识别、语义文本相似性、自然语言推断和问题回答等任务。我们提供了一个系统的研究，表明编码上下文信息的 LSTM 和句间交互至关重要，而 Tree-LSTM 并不能像先前宣传的那样有所帮助，但却出人意料地提高了 Twitter 数据集的性能；增强顺序推理模型是迄今为止较大数据集的最佳选择，而基于词对交互的模型在较少数据可用时实现最佳性能。我们将我们的实现作为开源工具包发布。

用于复述识别、语义文本相似性、自然语言推理和问答的神经网络模型

Neural Network Models for Paraphrase Identification, Semantic Textual  Similarity, Natural Language Inference, and Question Answering

Sentence pair modeling is critical for many NLP tasks, such as paraphrase
identification, semantic textual similarity, and natural language inference.
Most state-of-the-art neural models for these tasks rely on pretrained word
embedding and compose sentence-level semantics in varied ways; however, few
works have attempted to verify whether we really need pretrained embeddings in
these tasks. In this paper, we study how effective subword-level (character and
character n-gram) representations are in sentence pair modeling. Though it is
well-known that subword models are effective in tasks with single sentence
input, including language modeling and machine translation, they have not been
systematically studied in sentence pair modeling tasks where the semantic and
string similarities between texts matter. Our experiments show that subword
models without any pretrained word embedding can achieve new state-of-the-art
results on two social media datasets and competitive results on news data for
paraphrase identification.

该研究研究了如何使用基于子词级别表示的模型进行句子对建模，结果表明在对语义和字符串相似性要求较高的语境下，不需要预先训练的单词嵌入也可以实现最先进的结果。该方法在社交媒体数据集上实现了最新成果，在新闻数据集上也取得了有竞争力的结果。