This paper presents a novel model for multimodal learning based on gated
neural networks. The Gated Multimodal Unit (GMU) model is intended to be used
as an internal unit in a neural network architecture whose purpose is to find
an intermediate representation based on a combination of data from different
modalities. The GMU learns to decide how modalities influence the activation of
the unit using multiplicative gates. It was evaluated on a multilabel scenario
for genre classification of movies using the plot and the poster. The GMU
improved the macro f-score performance of single-modality approaches and
outperformed other fusion strategies, including mixture of experts models.
Along with this work, the MM-IMDb dataset is released which, to the best of our
knowledge, is the largest publicly available multimodal dataset for genre
prediction on movies.

本文介绍了一种基于门控神经网络的多模式学习新模型，称为 Gated Multimodal Unit (GMU) 模型，该模型用于在神经网络结构的内部单元中学习如何将来自不同模式的数据进行组合，并可用于电影类型分类中，通过使用剧情和电影海报的信息，模型在多标签情况下显著提高了单模式方法的性能，并优于其他的融合策略，同时提供了一个新的多模式数据集 MM-IMDb 用于电影类型分类的研究.