Variational autoencoders (VAE) combined with hierarchical RNNs have emerged
as a powerful framework for conversation modeling. However, they suffer from
the notorious degeneration problem, where the decoders learn to ignore latent
variables and reduce to vanilla RNNs. We empirically show that this degeneracy
occurs mostly due to two reasons. First, the expressive power of hierarchical
RNN decoders is often high enough to model the data using only its decoding
distributions without relying on the latent variables. Second, the conditional
VAE structure whose generation process is conditioned on a context, makes the
range of training targets very sparse; that is, the RNN decoders can easily
overfit to the training data ignoring the latent variables. To solve the
degeneration problem, we propose a novel model named Variational Hierarchical
Conversation RNNs (VHCR), involving two key ideas of (1) using a hierarchical
structure of latent variables, and (2) exploiting an utterance drop
regularization. With evaluations on two datasets of Cornell Movie Dialog and
Ubuntu Dialog Corpus, we show that our VHCR successfully utilizes latent
variables and outperforms state-of-the-art models for conversation generation.
Moreover, it can perform several new utterance control tasks, thanks to its
hierarchical latent structure.

通过使用层次潜在变量的结构和使用发言丢失正则化，我们提出一个名为 VHCR 的新模型，该模型成功地利用了潜在变量，并在会话生成方面优于现有技术。

一种变分对话建模的分层潜变结构

A Hierarchical Latent Structure for Variational Conversation Modeling

We present an approach that exploits hierarchical Recurrent Neural Networks
(RNNs) to tackle the video captioning problem, i.e., generating one or multiple
sentences to describe a realistic video. Our hierarchical framework contains a
sentence generator and a paragraph generator. The sentence generator produces
one simple short sentence that describes a specific short video interval. It
exploits both temporal- and spatial-attention mechanisms to selectively focus
on visual elements during generation. The paragraph generator captures the
inter-sentence dependency by taking as input the sentential embedding produced
by the sentence generator, combining it with the paragraph history, and
outputting the new initial state for the sentence generator. We evaluate our
approach on two large-scale benchmark datasets: YouTubeClips and
TACoS-MultiLevel. The experiments demonstrate that our approach significantly
outperforms the current state-of-the-art methods with BLEU@4 scores 0.499 and
0.305 respectively.

利用分层循环神经网络和时空注意力机制生成视频字幕，在 YouTube 和 TACoS 数据集上验证具有更高 BLEU 得分。

使用分层循环神经网络进行视频段落字幕生成

Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks

Existing deep convolutional neural networks (CNNs) have shown their great
success on image classification. CNNs mainly consist of convolutional and
pooling layers, both of which are performed on local image areas without
considering the dependencies among different image regions. However, such
dependencies are very important for generating explicit image representation.
In contrast, recurrent neural networks (RNNs) are well known for their ability
of encoding contextual information among sequential data, and they only require
a limited number of network parameters. General RNNs can hardly be directly
applied on non-sequential data. Thus, we proposed the hierarchical RNNs
(HRNNs). In HRNNs, each RNN layer focuses on modeling spatial dependencies
among image regions from the same scale but different locations. While the
cross RNN scale connections target on modeling scale dependencies among regions
from the same location but different scales. Specifically, we propose two
recurrent neural network models: 1) hierarchical simple recurrent network
(HSRN), which is fast and has low computational cost; and 2) hierarchical
long-short term memory recurrent network (HLSTM), which performs better than
HSRN with the price of more computational cost.
In this manuscript, we integrate CNNs with HRNNs, and develop end-to-end
convolutional hierarchical recurrent neural networks (C-HRNNs). C-HRNNs not
only make use of the representation power of CNNs, but also efficiently encodes
spatial and scale dependencies among different image regions. On four of the
most challenging object/scene image classification benchmarks, our C-HRNNs
achieve state-of-the-art results on Places 205, SUN 397, MIT indoor, and
competitive results on ILSVRC 2012.

该研究提出了一种基于深度卷积神经网络和分层递归神经网络的图像分类模型，以更好地编码图像区域之间的空间和比例依赖关系，从而取得了在四个具有挑战性的目标 / 场景图像分类基准测试中最先进的结果。