Tensor Attention, a multi-view attention that is able to capture high-order
correlations among multiple modalities, can overcome the representational
limitations of classical matrix attention. However, the $\Omega(n^3)$ time
complexity of tensor attention poses a significant obstacle to its practical
implementation in transformers, where $n$ is the input sequence length. In this
work, we prove that the backward gradient of tensor attention training can be
computed in almost linear $n^{1+o(1)}$ time, the same complexity as its forward
computation under a bounded entries assumption. We provide a closed-form
solution for the gradient and propose a fast computation method utilizing
polynomial approximation methods and tensor algebraic tricks. Furthermore, we
prove the necessity and tightness of our assumption through hardness analysis,
showing that slightly weakening it renders the gradient problem unsolvable in
truly subcubic time. Our theoretical results establish the feasibility of
efficient higher-order transformer training and may facilitate practical
applications of tensor attention architectures.

我们证明了张量注意力训练的反向梯度可以以几乎线性的 $n^{1+o (1)}$ 时间计算，同时提供了梯度的闭式解，并通过多项式逼近和张量代数技巧提出了一种快速计算方法。我们的理论结果证实了高阶 Transformer 训练的可行性，并可能促进张量注意力架构的实际应用。

张量注意力训练：高阶 Transformer 的可证明高效学习

Tensor Attention Training: Provably Efficient Learning of Higher-order  Transformers

Existing two-stream models for image-text matching show good performance
while ensuring retrieval speed and have received extensive attention from
industry and academia. These methods use a single representation to encode
image and text separately and get a matching score with cosine similarity or
the inner product of vectors. However, the performance of the two-stream model
is often sub-optimal. On the one hand, a single representation is challenging
to cover complex content comprehensively. On the other hand, in this framework
of lack of interaction, it is challenging to match multiple meanings which
leads to information being ignored. To address the problems mentioned above and
facilitate the performance of the two-stream model, we propose a multi-view
attention approach for two-stream image-text matching MVAM
(\textbf{M}ulti-\textbf{V}iew \textbf{A}ttention \textbf{M}odel). It first
learns multiple image and text representations by diverse attention heads with
different view codes. And then concatenate these representations into one for
matching. A diversity objective is also used to promote diversity between
attention heads. With this method, models are able to encode images and text
from different views and attend to more key points. So we can get
representations that contain more information. When doing retrieval tasks, the
matching scores between images and texts can be calculated from different
aspects, leading to better matching performance. Experiment results on MSCOCO
and Flickr30K show that our proposed model brings improvements over existing
models. Further case studies show that different attention heads can focus on
different contents and finally obtain a more comprehensive representation.

提出了一个针对两流图像 - 文本匹配的多视图注意力方法（MVAM），通过多样的注意头学习多个图像和文本表示，并将这些表示连接起来进行匹配，以实现更好的匹配性能和更全面的表示。

多视角关注的图像文本匹配

Image-Text Matching with Multi-View Attention

Answer selection and knowledge base question answering (KBQA) are two
important tasks of question answering (QA) systems. Existing methods solve
these two tasks separately, which requires large number of repetitive work and
neglects the rich correlation information between tasks. In this paper, we
tackle answer selection and KBQA tasks simultaneously via multi-task learning
(MTL), motivated by the following motivations. First, both answer selection and
KBQA can be regarded as a ranking problem, with one at text-level while the
other at knowledge-level. Second, these two tasks can benefit each other:
answer selection can incorporate the external knowledge from knowledge base
(KB), while KBQA can be improved by learning contextual information from answer
selection. To fulfill the goal of jointly learning these two tasks, we propose
a novel multi-task learning scheme that utilizes multi-view attention learned
from various perspectives to enable these tasks to interact with each other as
well as learn more comprehensive sentence representations. The experiments
conducted on several real-world datasets demonstrate the effectiveness of the
proposed method, and the performance of answer selection and KBQA is improved.
Also, the multi-view attention scheme is proved to be effective in assembling
attentive information from different representational perspectives.

本文使用多任务学习方法和多视角注意力机制，同时处理答案选择和知识库问答问题，并通过在不同视角学习多视角注意力来提高两个任务的性能。