Pre-trained seq2seq models have achieved state-of-the-art results in the
grammatical error correction task. However, these models still suffer from a
prediction bias due to their unidirectional decoding. Thus, we propose a
bidirectional Transformer reranker (BTR), that re-estimates the probability of
each candidate sentence generated by the pre-trained seq2seq model. The BTR
preserves the seq2seq-style Transformer architecture but utilizes a BERT-style
self-attention mechanism in the decoder to compute the probability of each
target token by using masked language modeling to capture bidirectional
representations from the target context. For guiding the reranking, the BTR
adopts negative sampling in the objective function to minimize the
unlikelihood. During inference, the BTR gives final results after comparing the
reranked top-1 results with the original ones by an acceptance threshold.
Experimental results show that, in reranking candidates from a pre-trained
seq2seq model, T5-base, the BTR on top of T5-base could yield 65.47 and 71.27
F0.5 scores on the CoNLL-14 and BEA test sets, respectively, and yield 59.52
GLEU score on the JFLEG corpus, with improvements of 0.36, 0.76 and 0.48 points
compared with the original T5-base. Furthermore, when reranking candidates from
T5-large, the BTR on top of T5-base improved the original T5-large by 0.26
points on the BEA test set.

通过采用 BERT 风格的自注意机制，我们提出了一种双向 Transformer 选手 (BTR)，可以找出由预训练 seq2seq 模型产生的候选句子中概率偏差的问题。与 T5-base 相比，BTR 在 CoNLL-14 和 BEA 测试集上可分别产生 65.47 和 71.27 F0.5 分数，在 JFLEG 语料库上可产生 59.52 GLEU 分数，优于 T5-base。

双向 Transformer 再排名用于语法错误纠正

Bidirectional Transformer Reranker for Grammatical Error Correction

A conventional approach to improving the performance of end-to-end speech
translation (E2E-ST) models is to leverage the source transcription via
pre-training and joint training with automatic speech recognition (ASR) and
neural machine translation (NMT) tasks. However, since the input modalities are
different, it is difficult to leverage source language text successfully. In
this work, we focus on sequence-level knowledge distillation (SeqKD) from
external text-based NMT models. To leverage the full potential of the source
language information, we propose backward SeqKD, SeqKD from a target-to-source
backward NMT model. To this end, we train a bilingual E2E-ST model to predict
paraphrased transcriptions as an auxiliary task with a single decoder. The
paraphrases are generated from the translations in bitext via back-translation.
We further propose bidirectional SeqKD in which SeqKD from both forward and
backward NMT models is combined. Experimental evaluations on both
autoregressive and non-autoregressive models show that SeqKD in each direction
consistently improves the translation performance, and the effectiveness is
complementary regardless of the model capacity.

本文提出了一种基于知识蒸馏的方法来提高端到端语音翻译模型的性能，并探讨了正反向模型、自回归和非自回归模型下的性能提升情况。

端到端语音翻译的源目标双向知识蒸馏

Source and Target Bidirectional Knowledge Distillation for End-to-end  Speech Translation

We introduce a new language representation model called BERT, which stands
for Bidirectional Encoder Representations from Transformers. Unlike recent
language representation models, BERT is designed to pre-train deep
bidirectional representations from unlabeled text by jointly conditioning on
both left and right context in all layers. As a result, the pre-trained BERT
model can be fine-tuned with just one additional output layer to create
state-of-the-art models for a wide range of tasks, such as question answering
and language inference, without substantial task-specific architecture
modifications.
BERT is conceptually simple and empirically powerful. It obtains new
state-of-the-art results on eleven natural language processing tasks, including
pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI
accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering
Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1
(5.1 point absolute improvement).

介绍了一种新的语言表示模型 BERT，可以通过预训练深度双向表示生成模型从未标记的文本中学习，通过微调可用于广泛的任务，包括自然语言处理。

BERT: 深度双向变换器的预训练用于语言理解

BERT: Pre-training of Deep Bidirectional Transformers for Language  Understanding

Aspect term extraction is one of the important subtasks in aspect-based
sentiment analysis. Previous studies have shown that using dependency tree
structure representation is promising for this task. However, most dependency
tree structures involve only one directional propagation on the dependency
tree. In this paper, we first propose a novel bidirectional dependency tree
network to extract dependency structure features from the given sentences. The
key idea is to explicitly incorporate both representations gained separately
from the bottom-up and top-down propagation on the given dependency syntactic
tree. An end-to-end framework is then developed to integrate the embedded
representations and BiLSTM plus CRF to learn both tree-structured and
sequential features to solve the aspect term extraction problem. Experimental
results demonstrate that the proposed model outperforms state-of-the-art
baseline models on four benchmark SemEval datasets.

本篇文章提出了一种深度学习模型，用于从给定句子中提取方面术语。 该模型基于双向依赖树，同时利用自下而上和自上而下的传播方法获得了更全面的信息，并且在 SemEval 数据集上优于现有的基准模型。