Video captioning aims to describe the content of videos using natural
language. Although significant progress has been made, there is still much room
to improve the performance for real-world applications, mainly due to the
long-tail words challenge. In this paper, we propose a text with knowledge
graph augmented transformer (TextKG) for video captioning. Notably, TextKG is a
two-stream transformer, formed by the external stream and internal stream. The
external stream is designed to absorb additional knowledge, which models the
interactions between the additional knowledge, e.g., pre-built knowledge graph,
and the built-in information of videos, e.g., the salient object regions,
speech transcripts, and video captions, to mitigate the long-tail words
challenge. Meanwhile, the internal stream is designed to exploit the
multi-modality information in videos (e.g., the appearance of video frames,
speech transcripts, and video captions) to ensure the quality of caption
results. In addition, the cross attention mechanism is also used in between the
two streams for sharing information. In this way, the two streams can help each
other for more accurate results. Extensive experiments conducted on four
challenging video captioning datasets, i.e., YouCookII, ActivityNet Captions,
MSRVTT, and MSVD, demonstrate that the proposed method performs favorably
against the state-of-the-art methods. Specifically, the proposed TextKG method
outperforms the best published results by improving 18.7% absolute CIDEr scores
on the YouCookII dataset.

本文提出了一种基于知识图谱的 transformer 方法 (TextKG)，分为内外两个流，外部流用于吸收额外知识、缓解长尾词等问题，内部流用于探索多模态信息以保证字幕结果质量。实验表明该方法在视觉字幕领域的四个数据集上均有较好表现。

知识图谱增强转换器用于视频字幕生成

Text with Knowledge Graph Augmented Transformer for Video Captioning

How to leverage dynamic contextual information in end-to-end speech
recognition has remained an active research area. Previous solutions to this
problem were either designed for specialized use cases that did not generalize
well to open-domain scenarios, did not scale to large biasing lists, or
underperformed on rare long-tail words. We address these limitations by
proposing a novel solution that combines shallow fusion, trie-based deep
biasing, and neural network language model contextualization. These techniques
result in significant 19.5% relative Word Error Rate improvement over existing
contextual biasing approaches and 5.4%-9.3% improvement compared to a strong
hybrid baseline on both open-domain and constrained contextualization tasks,
where the targets consist of mostly rare long-tail words. Our final system
remains lightweight and modular, allowing for quick modification without model
re-training.

提出了一种结合浅层融合、字典树和神经网络语言模型的方法，来利用动态的上下文信息，相比于已有的上下文偏置方法，这种方法最终的系统轻量且模块化，可以实现快速修改而无需重新训练，同时在开放域和约束化环境下都取得了显著的词错误率优化，特别是对于那些包含大部分稀有长尾词汇的语音转文字任务有着更高的表现。