Automatic video captioning aims for a holistic visual scene understanding. It
requires a mechanism for capturing temporal context in video frames and the
ability to comprehend the actions and associations of objects in a given
timeframe. Such a system should additionally learn to abstract video sequences
into sensible representations as well as to generate natural written language.
While the majority of captioning models focus solely on the visual inputs,
little attention has been paid to the audiovisual modality. To tackle this
issue, we propose a novel two-fold approach. First, we implement a
reward-guided KL Divergence to train a video captioning model which is
resilient towards token permutations. Second, we utilise a Bi-Modal
Hierarchical Reinforcement Learning (BMHRL) Transformer architecture to capture
long-term temporal dependencies of the input data as a foundation for our
hierarchical captioning module. Using our BMHRL, we show the suitability of the
HRL agent in the generation of content-complete and grammatically sound
sentences by achieving $4.91$, $2.23$, and $10.80$ in BLEU3, BLEU4, and METEOR
scores, respectively on the ActivityNet Captions dataset. Finally, we make our
BMHRL framework and trained models publicly available for users and developers
at this https URL.

提出了一种新颖的双重方法，BMHRL，用于自动视频字幕。使用 BMHRL，该研究展示了基于 HRL 代理的内容完整和语法良好的语句生成，并为用户和开发人员提供了 BMHRL 框架和训练模型的公开功能。