A visual hard attention model actively selects and observes a sequence of
subregions in an image to make a prediction. The majority of hard attention
models determine the attention-worthy regions by first analyzing a complete
image. However, it may be the case that the entire image is not available
initially but instead sensed gradually through a series of partial
observations. In this paper, we design an efficient hard attention model for
classifying such sequentially observed scenes. The presented model never
observes an image completely. To select informative regions under partial
observability, the model uses Bayesian Optimal Experiment Design. First, it
synthesizes the features of the unobserved regions based on the already
observed regions. Then, it uses the predicted features to estimate the expected
information gain (EIG) attained, should various regions be attended. Finally,
the model attends to the actual content on the location where the EIG mentioned
above is maximum. The model uses a) a recurrent feature aggregator to maintain
a recurrent state, b) a linear classifier to predict the class label, c) a
Partial variational autoencoder to predict the features of unobserved regions.
We use normalizing flows in Partial VAE to handle multi-modality in the
feature-synthesis problem. We train our model using a differentiable objective
and test it on five datasets. Our model gains 2-10% higher accuracy than the
baseline models when both have seen only a couple of glimpses.

本文阐述了一种利用 Bayesian Optimal Experiment Design 来选择信息区域的视觉 Hard Attention Model。通过合成未观察到的区域的特征来预测信息增益（EIG），并在 EIG 最大的位置上实际观察内容。这种模型使用递归特征聚合器维护递归状态，使用线性分类器来预测类别标签，并使用部分变分自编码器来预测未观察到区域的特征。我们使用可微分的目标函数对模型进行训练，并在五个数据集上进行测试。与基线模型相比，当两者只看到几个方向时，我们的模型获得了 2-10％更高的准确度。