Image description generation plays an important role in many real-world
applications, such as image retrieval, automatic navigation, and disabled
people support. A well-developed task of image description generation is image
captioning, which usually generates a short captioning sentence and thus
neglects many of fine-grained properties, e.g., the information of subtle
objects and their relationships. In this paper, we study the visual paragraph
generation, which can describe the image with a long paragraph containing rich
details. Previous research often generates the paragraph via a hierarchical
Recurrent Neural Network (RNN)-like model, which has complex memorising,
forgetting and coupling mechanism. Instead, we propose a novel pure CNN model,
ParaCNN, to generate visual paragraph using hierarchical CNN architecture with
contextual information between sentences within one paragraph. The ParaCNN can
generate an arbitrary length of a paragraph, which is more applicable in many
real-world applications. Furthermore, to enable the ParaCNN to model paragraph
comprehensively, we also propose an adversarial twin net training scheme.
During training, we force the forwarding network's hidden features to be close
to that of the backwards network by using adversarial training. During testing,
we only use the forwarding network, which already includes the knowledge of the
backwards network, to generate a paragraph. We conduct extensive experiments on
the Stanford Visual Paragraph dataset and achieve state-of-the-art performance.

本文提出了一种使用纯 CNN 模型生成可包含丰富细节的长段落视觉描述的方法，使用上下文信息的层次 CNN 架构，以及对抗性训练方案，以实现全面建模段落的目的，并在 Stanford 视觉段落数据集上获得了最先进的性能。

ParaCNN: 基于对抗孪生上下文 CNN 的视觉段落生成

ParaCNN: Visual Paragraph Generation via Adversarial Twin Contextual  CNNs

Visual paragraph generation aims to automatically describe a given image from
different perspectives and organize sentences in a coherent way. In this paper,
we address three critical challenges for this task in a reinforcement learning
setting: the mode collapse, the delayed feedback, and the time-consuming
warm-up for policy networks. Generally, we propose a novel Curiosity-driven
Reinforcement Learning (CRL) framework to jointly enhance the diversity and
accuracy of the generated paragraphs. First, by modeling the paragraph
captioning as a long-term decision-making process and measuring the prediction
uncertainty of state transitions as intrinsic rewards, the model is
incentivized to memorize precise but rarely spotted descriptions to context,
rather than being biased towards frequent fragments and generic patterns.
Second, since the extrinsic reward from evaluation is only available until the
complete paragraph is generated, we estimate its expected value at each time
step with temporal-difference learning, by considering the correlations between
successive actions. Then the estimated extrinsic rewards are complemented by
dense intrinsic rewards produced from the derived curiosity module, in order to
encourage the policy to fully explore action space and find a global optimum.
Third, discounted imitation learning is integrated for learning from human
demonstrations, without separately performing the time-consuming warm-up in
advance. Extensive experiments conducted on the Standford image-paragraph
dataset demonstrate the effectiveness and efficiency of the proposed method,
improving the performance by 38.4% compared with state-of-the-art.

本研究提出一种基于好奇心驱动的强化学习框架（CRL），旨在通过在长期决策过程中测量状态转移预测不确定性、引入由好奇心模块产生的密集的固有奖励、以及整合折扣模仿学习等方法，共同提高视觉段落生成的多样性和准确性