Attention models are typically learned by optimizing one of three standard
loss functions that are variously called -- soft attention, hard attention, and
latent variable marginal likelihood (LVML) attention. All three paradigms are
motivated by the same goal of finding two models -- a `focus' model that
`selects' the right \textit{segment} of the input and a `classification' model
that processes the selected segment into the target label. However, they differ
significantly in the way the selected segments are aggregated, resulting in
distinct dynamics and final results. We observe a unique signature of models
learned using these paradigms and explain this as a consequence of the
evolution of the classification model under gradient descent when the focus
model is fixed. We also analyze these paradigms in a simple setting and derive
closed-form expressions for the parameter trajectory under gradient flow. With
the soft attention loss, the focus model improves quickly at initialization and
splutters later on. On the other hand, hard attention loss behaves in the
opposite fashion. Based on our observations, we propose a simple hybrid
approach that combines the advantages of the different loss functions and
demonstrates it on a collection of semi-synthetic and real-world datasets

关注模型有三种不同的潜在变量边际可能性（LVML）：软关注、硬关注和软关注损失。我们观察到使用这些范例学到的模型具有独特的特征，并且提出了一种简单的混合方法，结合了不同损失函数的优点。