Generative models have recently started to outperform extractive models in
Open Domain Question Answering, largely by leveraging their decoder to attend
over multiple encoded passages and combining their information. However,
generative models tend to be larger than extractive models due to the need for
a decoder, run slower during inference due to auto-regressive decoder beam
search, and their generated output often suffers from hallucinations. We
propose to extend transformer encoders with the ability to fuse information
from multiple passages, using global representation to provide cross-sample
attention over all tokens across samples. Furthermore, we propose an
alternative answer span probability calculation to better aggregate answer
scores in the global space of all samples. Using our proposed method, we
outperform the current state-of-the-art method by $2.5$ Exact Match score on
the Natural Question dataset while using only $25\%$ of parameters and $35\%$
of the latency during inference, and $4.4$ Exact Match on WebQuestions dataset.
When coupled with synthetic data augmentation, we outperform larger models on
the TriviaQA dataset as well. The latency and parameter savings of our method
make it particularly attractive for open-domain question answering, as these
models are often compute-intensive.

通过扩展 Transformer 编码器的能力以融合信息，使用全局表示跨样本注意所有令牌，并提出了一种更好的答案跨度概率计算方法，我们使用与综合数据增强配对的方法，在推理期间只使用 25％的参数和 35％的延迟，在自然问题数据集上的精确匹配分数达到了当前技术水平的 2.5 和 WebQuestions 数据集的 4.4。这种方法的延迟和参数节省特别适用于计算密集型应用，例如开放域问题回答。

利用编码器早期融合创建全球概率空间以进行开放领域问答

FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering

For video recognition task, a global representation summarizing the whole
contents of the video snippets plays an important role for the final
performance. However, existing video architectures usually generate it by using
a simple, global average pooling (GAP) method, which has limited ability to
capture complex dynamics of videos. For image recognition task, there exist
evidences showing that covariance pooling has stronger representation ability
than GAP. Unfortunately, such plain covariance pooling used in image
recognition is an orderless representative, which cannot model spatio-temporal
structure inherent in videos. Therefore, this paper proposes a
Temporal-attentive Covariance Pooling(TCP), inserted at the end of deep
architectures, to produce powerful video representations. Specifically, our TCP
first develops a temporal attention module to adaptively calibrate
spatio-temporal features for the succeeding covariance pooling, approximatively
producing attentive covariance representations. Then, a temporal covariance
pooling performs temporal pooling of the attentive covariance representations
to characterize both intra-frame correlations and inter-frame
cross-correlations of the calibrated features. As such, the proposed TCP can
capture complex temporal dynamics. Finally, a fast matrix power normalization
is introduced to exploit geometry of covariance representations. Note that our
TCP is model-agnostic and can be flexibly integrated into any video
architectures, resulting in TCPNet for effective video recognition. The
extensive experiments on six benchmarks (e.g., Kinetics, Something-Something V1
and Charades) using various video architectures show our TCPNet is clearly
superior to its counterparts, while having strong generalization ability. The
source code is publicly available.

本文提出了一种基于时间注意力和内帧关联的计算方式 ——Temporal-attentive Covariance Pooling (TCP)，它可以灵活地集成到任何视频架构中，用于生成强大的视频表示，它在六个基准测试中都表现出优异的性能，并具有强大的泛化能力。

面向视频识别的时序关注协方差汇聚网络

Temporal-attentive Covariance Pooling Networks for Video Recognition

Transformer, based on the encoder-decoder framework, has achieved
state-of-the-art performance on several natural language generation tasks. The
encoder maps the words in the input sentence into a sequence of hidden states,
which are then fed into the decoder to generate the output sentence. These
hidden states usually correspond to the input words and focus on capturing
local information. However, the global (sentence level) information is seldom
explored, leaving room for the improvement of generation quality. In this
paper, we propose a novel global representation enhanced Transformer (GRET) to
explicitly model global representation in the Transformer network.
Specifically, in the proposed model, an external state is generated for the
global representation from the encoder. The global representation is then fused
into the decoder during the decoding process to improve generation quality. We
conduct experiments in two text generation tasks: machine translation and text
summarization. Experimental results on four WMT machine translation tasks and
LCSTS text summarization task demonstrate the effectiveness of the proposed
approach on natural language generation.

本文提出了一种在 Transformer 网络中显式地建模全局表示的新方法 —— 全局表示增强的 Transformer（GRET），并将其应用于机器翻译和文本摘要两个文本生成任务中，并通过实验证明其在自然语言生成方面的有效性。