We describe a DNN for video classification and captioning, trained
end-to-end, with shared features, to solve tasks at different levels of
granularity, exploring the link between granularity in a source task and the
quality of learned features for transfer learning. For solving the new task
domain in transfer learning, we freeze the trained encoder and fine-tune a
neural net on the target domain. We train on the Something-Something dataset
with over 220, 000 videos, and multiple levels of target granularity, including
50 action groups, 174 fine-grained action categories and captions.
Classification and captioning with Something-Something are challenging because
of the subtle differences between actions, applied to thousands of different
object classes, and the diversity of captions penned by crowd actors. Our model
performs better than existing classification baselines for SomethingSomething,
with impressive fine-grained results. And it yields a strong baseline on the
new Something-Something captioning task. Experiments reveal that training with
more fine-grained tasks tends to produce better features for transfer learning.

本文提出了一个 DNN 模型，用于视频分类和字幕生成，它经过端到端的训练，在不同粒度任务中共享特征，在迁移学习中获得更好的表现。在 Something-Something 数据集上，该模型比现有的分类基线具有更好的性能，并在新的 Something-Something 字幕生成任务中提供了强有力的基准。实验表明，通过训练更多的细粒度任务，可以产生更好的特征用于迁移学习。

任务粒度对迁移学习的有效性

On the effectiveness of task granularity for transfer learning

The "CNN-RNN" design pattern is increasingly widely applied in a variety of
image annotation tasks including multi-label classification and captioning.
Existing models use the weakly semantic CNN hidden layer or its transform as
the image embedding that provides the interface between the CNN and RNN. This
leaves the RNN overstretched with two jobs: predicting the visual concepts and
modelling their correlations for generating structured annotation output.
Importantly this makes the end-to-end training of the CNN and RNN slow and
ineffective due to the difficulty of back propagating gradients through the RNN
to train the CNN. We propose a simple modification to the design pattern that
makes learning more effective and efficient. Specifically, we propose to use a
semantically regularised embedding layer as the interface between the CNN and
RNN. Regularising the interface can partially or completely decouple the
learning problems, allowing each to be more effectively trained and jointly
training much more efficient. Extensive experiments show that state-of-the art
performance is achieved on multi-label classification as well as image
captioning.

本文研究为了改进多标签分类与图像标注任务中 CNN-RNN 设计模式，在 CNN 与 RNN 之间引入一个有着语义规范化的嵌入层，以分离学习任务并使得训练更加高效。实验证明，该模式取得了多标签分类与图像标注任务的最新成果。

循环图片注释的语义正则化

Semantic Regularisation for Recurrent Image Annotation

We propose a simple domain adaptation method for neural networks in a
supervised setting. Supervised domain adaptation is a way of improving the
generalization performance on the target domain by using the source domain
dataset, assuming that both of the datasets are labeled. Recently, recurrent
neural networks have been shown to be successful on a variety of NLP tasks such
as caption generation; however, the existing domain adaptation techniques are
limited to (1) tune the model parameters by the target dataset after the
training by the source dataset, or (2) design the network to have dual output,
one for the source domain and the other for the target domain. Reformulating
the idea of the domain adaptation technique proposed by Daume (2007), we
propose a simple domain adaptation method, which can be applied to neural
networks trained with a cross-entropy loss. On captioning datasets, we show
performance improvements over other domain adaptation methods.

本文提出了一种简单的领域自适应方法，可以应用于使用交叉熵损失训练的神经网络，在字幕数据集上展示了该方法相对于其他领域自适应方法的表现提高。

参数增强的神经网络域适应

Domain Adaptation for Neural Networks by Parameter Augmentation

We propose a novel extension of the encoder-decoder framework, called a
review network. The review network is generic and can enhance any existing
encoder- decoder model: in this paper, we consider RNN decoders with both CNN
and RNN encoders. The review network performs a number of review steps with
attention mechanism on the encoder hidden states, and outputs a thought vector
after each review step; the thought vectors are used as the input of the
attention mechanism in the decoder. We show that conventional encoder-decoders
are a special case of our framework. Empirically, we show that our framework
improves over state-of- the-art encoder-decoder systems on the tasks of image
captioning and source code captioning.

该研究提出了一种扩展的编码器 - 解码器框架 - 评论网络，通过对编码器隐藏状态进行多次注意力机制的评论步骤来输出思维向量，并将这些思维向量作为解码器中的注意力机制的输入，该框架在图像标题和源代码标题两个任务上表现优于现有的编码器 - 解码器系统。