As the most critical components in a sentence, subject, predicate and object
require special attention in the video captioning task. To implement this idea,
we design a novel framework, named COllaborative three-Stream Transformers
(COST), to model the three parts separately and complement each other for
better representation. Specifically, COST is formed by three branches of
transformers to exploit the visual-linguistic interactions of different
granularities in spatial-temporal domain between videos and text, detected
objects and text, and actions and text. Meanwhile, we propose a
cross-granularity attention module to align the interactions modeled by the
three branches of transformers, then the three branches of transformers can
support each other to exploit the most discriminative semantic information of
different granularities for accurate predictions of captions. The whole model
is trained in an end-to-end fashion. Extensive experiments conducted on three
large-scale challenging datasets, i.e., YouCookII, ActivityNet Captions and
MSVD, demonstrate that the proposed method performs favorably against the
state-of-the-art methods.

通过设计一个名为 COllaborative three-Stream Transformers（COST）的新框架，来在视频字幕任务中对句子的主语、谓语和宾语进行特殊关注，通过三个分支的 transformers 以及交叉粒度的注意力模块对不同粒度的视觉 - 语言交互进行建模和对齐，以获得准确的字幕预测。实验证明，COST 方法在视频字幕领域中表现优于现有的方法。

协作三流变压器用于视频字幕生成

Collaborative Three-Stream Transformers for Video Captioning

In typical neural machine translation~(NMT), the decoder generates a sentence
word by word, packing all linguistic granularities in the same time-scale of
RNN. In this paper, we propose a new type of decoder for NMT, which splits the
decode state into two parts and updates them in two different time-scales.
Specifically, we first predict a chunk time-scale state for phrasal modeling,
on top of which multiple word time-scale states are generated. In this way, the
target sentence is translated hierarchically from chunks to words, with
information in different granularities being leveraged. Experiments show that
our proposed model significantly improves the translation performance over the
state-of-the-art NMT model.

本文提出了一种新型神经机器翻译解码器，通过将解码状态分成两个部分并在不同时间尺度上更新，实现从词块到单词的分级翻译，从而提高了翻译性能。