Sentence scoring aims at measuring the likelihood score of a sentence and is
widely used in many natural language processing scenarios, like reranking,
which is to select the best sentence from multiple candidates. Previous works
on sentence scoring mainly adopted either causal language modeling (CLM) like
GPT or masked language modeling (MLM) like BERT, which have some limitations:
1) CLM only utilizes unidirectional information for the probability estimation
of a sentence without considering bidirectional context, which affects the
scoring quality; 2) MLM can only estimate the probability of partial tokens at
a time and thus requires multiple forward passes to estimate the probability of
the whole sentence, which incurs large computation and time cost. In this
paper, we propose \textit{Transcormer} -- a Transformer model with a novel
\textit{sliding language modeling} (SLM) for sentence scoring. Specifically,
our SLM adopts a triple-stream self-attention mechanism to estimate the
probability of all tokens in a sentence with bidirectional context and only
requires a single forward pass. SLM can avoid the limitations of CLM (only
unidirectional context) and MLM (multiple forward passes) and inherit their
advantages, and thus achieve high effectiveness and efficiency in scoring.
Experimental results on multiple tasks demonstrate that our method achieves
better performance than other language modelings.

本文提出了一个基于 Transformer 的滑动语言建模技术来进行句子评分，该方法可以避免传统方法只考虑单向上下文、多次前向传递带来的计算代价等问题，并在多项任务中展现出更好的效率和准确度。

Transcormer：滑动语言建模句子评分的 Transformer

Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling

In automatic speech recognition, many studies have shown performance
improvements using language models (LMs). Recent studies have tried to use
bidirectional LMs (biLMs) instead of conventional unidirectional LMs (uniLMs)
for rescoring the $N$-best list decoded from the acoustic model. In spite of
their theoretical benefits, the biLMs have not given notable improvements
compared to the uniLMs in their experiments. This is because their biLMs do not
consider the interaction between the two directions. In this paper, we propose
a novel sentence scoring method considering the interaction between the past
and the future words on the biLM. Our experimental results on the LibriSpeech
corpus show that the biLM with the proposed sentence scoring outperforms the
uniLM for the $N$-best list rescoring, consistently and significantly in all
experimental conditions. The analysis of WERs by word position demonstrates
that the biLM is more robust than the uniLM especially when a recognized
sentence is short or a misrecognized word is at the beginning of the sentence.

本文提出了一种考虑 biLM 过去和未来单词之间交互的句子得分方法，在对 N 个最佳列表重新评分时，biLM 显著优于 uniLM，特别是在识别的句子很短或错识单词位于句子开头时。

利用双向语言模型进行有效的语句评分方法，用于语音识别

Effective Sentence Scoring Method using Bidirectional Language Model for  Speech Recognition

Sentence scoring and sentence selection are two main steps in extractive
document summarization systems. However, previous works treat them as two
separated subtasks. In this paper, we present a novel end-to-end neural network
framework for extractive document summarization by jointly learning to score
and select sentences. It first reads the document sentences with a hierarchical
encoder to obtain the representation of sentences. Then it builds the output
summary by extracting sentences one by one. Different from previous methods,
our approach integrates the selection strategy into the scoring model, which
directly predicts the relative importance given previously selected sentences.
Experiments on the CNN/Daily Mail dataset show that the proposed framework
significantly outperforms the state-of-the-art extractive summarization models.

在这篇论文中，我们提出了一种新颖的端到端神经网络框架，通过联合学习对句子进行评分和选择来实现摘要提取。在 CNN / Daily Mail 数据集上的实验证明，我们的方法比现有的抽取式摘要模型表现出更好的效果。