We propose "Areas of Attention", a novel attention-based model for automatic
image captioning. Our approach models the dependencies between image regions,
caption words, and the state of an RNN language model, using three pairwise
interactions. In contrast to previous attention-based approaches that associate
image regions only to the RNN state, our method allows a direct association
between caption words and image regions. During training these associations are
inferred from image-level captions, akin to weakly-supervised object detector
training. These associations help to improve captioning by localizing the
corresponding regions during testing. We also propose and compare different
ways of generating attention areas: CNN activation grids, object proposals, and
spatial transformers nets applied in a convolutional fashion. Spatial
transformers give the best results. They allow for image specific attention
areas, and can be trained jointly with the rest of the network. Our attention
mechanism and spatial transformer attention areas together yield
state-of-the-art results on the MSCOCO dataset.o meaningful latent semantic
structure in the generated captions.

提出了基于注意力机制的图像描述模型，该模型使用三种成对的交互来建立图像区域、描述文本和 RNN 语言模型之间的依赖关系，并运用空间变换器来生成最佳的区域关注机制，达到了 MSCOCO 数据集上的最佳结果。

图像字幕注意力区域

Areas of Attention for Image Captioning

In this paper, we propose a generative knowledge transfer technique that
trains an RNN based language model (student network) using text and output
probabilities generated from a previously trained RNN (teacher network). The
text generation can be conducted by either the teacher or the student network.
We can also improve the performance by taking the ensemble of soft labels
obtained from multiple teacher networks. This method can be used for privacy
conscious language model adaptation because no user data is directly used for
training. Especially, when the soft labels of multiple devices are aggregated
via a trusted third party, we can expect very strong privacy protection.

本文提出了一种生成式知识迁移的技术，使用以前训练的基于 RNN 的语言模型 (RNN 学生网络) 生成的文本和输出概率来训练一个新的语言模型。我们还可以通过使用多个教师网络的软标签组合来提高性能，从而适用于隐私意识的语言模型自适应，因为不会直接使用用户数据进行训练。特别是，当多个设备的软标签通过可信的第三方进行聚合时，我们可以期望非常强的隐私保护。

神经语言模型的生成式知识转移

Generative Knowledge Transfer for Neural Language Models

Current language models have a significant limitation in the ability to
encode and decode factual knowledge. This is mainly because they acquire such
knowledge from statistical co-occurrences although most of the knowledge words
are rarely observed. In this paper, we propose a Neural Knowledge Language
Model (NKLM) which combines symbolic knowledge provided by the knowledge graph
with the RNN language model. By predicting whether the word to generate has an
underlying fact or not, the model can generate such knowledge-related words by
copying from the description of the predicted fact. In experiments, we show
that the NKLM significantly improves the performance while generating a much
smaller number of unknown words.

该研究提出了一种神经知识语言模型 (NKLM)，将知识图提供的符号化知识与 RNN 语言模型相结合，通过预测所生成的单词是否具有潜在事实，从预测事实的描述中复制生成这些与知识相关的单词。实验结果表明，NKLM 显著提高了性能的同时，生成了更少的未知词。