This paper presents a novel CNN-RNN based approach, which exploits multiple
CNN features for dimensional emotion recognition in-the-wild, utilizing the
One-Minute Gradual-Emotion (OMG-Emotion) dataset. Our approach includes first
pre-training with the relevant and large in size, Aff-Wild and Aff-Wild2
emotion databases. Low-, mid- and high-level features are extracted from the
trained CNN component and are exploited by RNN subnets in a multi-task
framework. Their outputs constitute an intermediate level prediction; final
estimates are obtained as the mean or median values of these predictions.
Fusion of the networks is also examined for boosting the obtained performance,
at Decision-, or at Model-level; in the latter case a RNN was used for the
fusion. Our approach, although using only the visual modality, outperformed
state-of-the-art methods that utilized audio and visual modalities. Some of our
developments have been submitted to the OMG-Emotion Challenge, ranking second
among the technologies which used only visual information for valence
estimation; ranking third overall. Through extensive experimentation, we
further show that arousal estimation is greatly improved when low-level
features are combined with high-level ones.

本文利用 CNN-RNN 方法以及 OMG-Emotion 数据集，多模态特征提取，提出了情感识别的新思路，仅利用视觉模态的方法优于同时利用音频和视觉模态的现有技术，实验表明，低级特征结合高级特征可明显提高情感唤起度的识别。

基于多 CNN 特征的 CNN-RNN 情感维度识别在 OMG 野外数据集上的应用

Exploiting multi-CNN features in CNN-RNN based Dimensional Emotion  Recognition on the OMG in-the-wild Dataset

The "CNN-RNN" design pattern is increasingly widely applied in a variety of
image annotation tasks including multi-label classification and captioning.
Existing models use the weakly semantic CNN hidden layer or its transform as
the image embedding that provides the interface between the CNN and RNN. This
leaves the RNN overstretched with two jobs: predicting the visual concepts and
modelling their correlations for generating structured annotation output.
Importantly this makes the end-to-end training of the CNN and RNN slow and
ineffective due to the difficulty of back propagating gradients through the RNN
to train the CNN. We propose a simple modification to the design pattern that
makes learning more effective and efficient. Specifically, we propose to use a
semantically regularised embedding layer as the interface between the CNN and
RNN. Regularising the interface can partially or completely decouple the
learning problems, allowing each to be more effectively trained and jointly
training much more efficient. Extensive experiments show that state-of-the art
performance is achieved on multi-label classification as well as image
captioning.

本文研究为了改进多标签分类与图像标注任务中 CNN-RNN 设计模式，在 CNN 与 RNN 之间引入一个有着语义规范化的嵌入层，以分离学习任务并使得训练更加高效。实验证明，该模式取得了多标签分类与图像标注任务的最新成果。

循环图片注释的语义正则化

Semantic Regularisation for Recurrent Image Annotation

Deep neural network (DNN) models have recently obtained state-of-the-art
prediction accuracy for the transcription factor binding (TFBS) site
classification task. However, it remains unclear how these approaches identify
meaningful DNA sequence signals and give insights as to why TFs bind to certain
locations. In this paper, we propose a toolkit called the Deep Motif Dashboard
(DeMo Dashboard) which provides a suite of visualization strategies to extract
motifs, or sequence patterns from deep neural network models for TFBS
classification. We demonstrate how to visualize and understand three important
DNN models: convolutional, recurrent, and convolutional-recurrent networks. Our
first visualization method is finding a test sequence's saliency map which uses
first-order derivatives to describe the importance of each nucleotide in making
the final prediction. Second, considering recurrent models make predictions in
a temporal manner (from one end of a TFBS sequence to the other), we introduce
temporal output scores, indicating the prediction score of a model over time
for a sequential input. Lastly, a class-specific visualization strategy finds
the optimal input sequence for a given TFBS positive class via stochastic
gradient optimization. Our experimental results indicate that a
convolutional-recurrent architecture performs the best among the three
architectures. The visualization techniques indicate that CNN-RNN makes
predictions by modeling both motifs as well as dependencies among them.

本文提出了一个名为 Deep Motif Dashboard 的工具套件，提供了一系列深度学习网络模型中 DNA 定序信号及转录因子结合位点分类的可视化策略。其中，卷积递归架构在实验结果中表现最佳，其可视化技术指出了 CNN-RNN 模型在建模模式以及其中相互依赖的模体方面的性能。