We propose a high-level concept word detector that can be integrated with any
video-to-language models. It takes a video as input and generates a list of
concept words as useful semantic priors for language generation models. The
proposed word detector has two important properties. First, it does not require
any external knowledge sources for training. Second, the proposed word detector
is trainable in an end-to-end manner jointly with any video-to-language models.
To maximize the values of detected words, we also develop a semantic attention
mechanism that selectively focuses on the detected concept words and fuse them
with the word encoding and decoding in the language model. In order to
demonstrate that the proposed approach indeed improves the performance of
multiple video-to-language tasks, we participate in four tasks of LSMDC 2016.
Our approach achieves the best accuracies in three of them, including
fill-in-the-blank, multiple-choice test, and movie retrieval. We also attain
comparable performance for the other task, movie description.

本文提出了一种高级概念词探测器，可以与任何语言生成模型集成。它以视频为输入，并生成一个概念词清单作为语言生成模型的有用语义先验。所提出的词探测器有两个重要特征。它不需要任何外部知识源进行训练；它可以与任何视频到语言模型联合进行端对端的训练。为了最大化检测到的词的价值，我们还开发了一种语义关注机制，它有选择性地聚焦于检测到的概念词，并将它们与语言模型中的词编码和译码融合。为了证明所提出的方法确实改善了多个视频到语言任务的性能，我们参与了 LSMDC 2016 的四个任务。我们的方法在其中三个任务中实现了最佳准确度，包括填空、多项选择测试和电影检索。我们还获得了其他任务（电影描述）的可比性能。