Image captioning is a research hotspot where encoder-decoder models combining
convolutional neural network (CNN) and long short-term memory (LSTM) achieve
promising results. Despite significant progress, these models generate
sentences differently from human cognitive styles. Existing models often
generate a complete sentence from the first word to the end, without
considering the influence of the following words on the whole sentence
generation. In this paper, we explore the utilization of a human-like cognitive
style, i.e., building overall cognition for the image to be described and the
sentence to be constructed, for enhancing computer image understanding. This
paper first proposes a Mutual-aid network structure with Bidirectional LSTMs
(MaBi-LSTMs) for acquiring overall contextual information. In the training
process, the forward and backward LSTMs encode the succeeding and preceding
words into their respective hidden states by simultaneously constructing the
whole sentence in a complementary manner. In the captioning process, the LSTM
implicitly utilizes the subsequent semantic information contained in its hidden
states. In fact, MaBi-LSTMs can generate two sentences in forward and backward
directions. To bridge the gap between cross-domain models and generate a
sentence with higher quality, we further develop a cross-modal attention
mechanism to retouch the two sentences by fusing their salient parts as well as
the salient areas of the image. Experimental results on the Microsoft COCO
dataset show that the proposed model improves the performance of
encoder-decoder models and achieves state-of-the-art results.

本文探讨利用类人认知风格，即建立对要描述的图像和构建的句子的整体认知来提高计算机图像理解能力。针对现有模型生成句子时没有考虑后续单词对整个句子生成的影响，本文首先提出了一种利用双向 LSTM 的相互援助网络结构（MaBi-LSTMs），以获取整体上下文信息。其次，为了提高跨领域模型生成更高质量的句子，我们进一步开发了跨模态注意机制来修饰两个句子，通过融合其显著部分以及图像的显著区域。在 Microsoft COCO 数据集上的实验结果表明，该模型改善了编码器解码器模型的性能，并取得了最先进的结果。