It is always well believed that parsing an image into constituent visual
patterns would be helpful for understanding and representing an image.
Nevertheless, there has not been evidence in support of the idea on describing
an image with a natural-language utterance. In this paper, we introduce a new
design to model a hierarchy from instance level (segmentation), region level
(detection) to the whole image to delve into a thorough image understanding for
captioning. Specifically, we present a HIerarchy Parsing (HIP) architecture
that novelly integrates hierarchical structure into image encoder. Technically,
an image decomposes into a set of regions and some of the regions are resolved
into finer ones. Each region then regresses to an instance, i.e., foreground of
the region. Such process naturally builds a hierarchal tree. A tree-structured
Long Short-Term Memory (Tree-LSTM) network is then employed to interpret the
hierarchal structure and enhance all the instance-level, region-level and
image-level features. Our HIP is appealing in view that it is pluggable to any
neural captioning models. Extensive experiments on COCO image captioning
dataset demonstrate the superiority of HIP. More remarkably, HIP plus a
top-down attention-based LSTM decoder increases CIDEr-D performance from 120.1%
to 127.2% on COCO Karpathy test split. When further endowing instance-level and
region-level features from HIP with semantic relation learnt through Graph
Convolutional Networks (GCN), CIDEr-D is boosted up to 130.6%.

该论文提出了一种基于分层结构的图像编码器，通过将图像分解为实例级别、区域级别和整体级别的层次结构，引入了分层解析（HIP）架构，大大提高了基于神经网络的图像描述性能，结合图卷积网络（GCN）可以进一步提高描述性能。