We propose an Auto-Parsing Network (APN) to discover and exploit the input
data's hidden tree structures for improving the effectiveness of the
Transformer-based vision-language systems. Specifically, we impose a
Probabilistic Graphical Model (PGM) parameterized by the attention operations
on each self-attention layer to incorporate sparse assumption. We use this PGM
to softly segment an input sequence into a few clusters where each cluster can
be treated as the parent of the inside entities. By stacking these PGM
constrained self-attention layers, the clusters in a lower layer compose into a
new sequence, and the PGM in a higher layer will further segment this sequence.
Iteratively, a sparse tree can be implicitly parsed, and this tree's
hierarchical knowledge is incorporated into the transformed embeddings, which
can be used for solving the target vision-language tasks. Specifically, we
showcase that our APN can strengthen Transformer based networks in two major
vision-language tasks: Captioning and Visual Question Answering. Also, a PGM
probability-based parsing algorithm is developed by which we can discover what
the hidden structure of input is during the inference.

提出了一种自动解析网络（APN），将概率图像模型（PGM）参数化为自注意层上的注意操作，以将稀疏假设合并到输入序列的分段中，从而提高基于 Transformer 的视觉语言系统的效率。