Multi-label image and video classification are fundamental yet challenging
tasks in computer vision. The main challenges lie in capturing spatial or
temporal dependencies between labels and discovering the locations of
discriminative features for each class. In order to overcome these challenges,
we propose to use cross-modality attention with semantic graph embedding for
multi label classification. Based on the constructed label graph, we propose an
adjacency-based similarity graph embedding method to learn semantic label
embeddings, which explicitly exploit label relationships. Then our novel
cross-modality attention maps are generated with the guidance of learned label
embeddings. Experiments on two multi-label image classification datasets
(MS-COCO and NUS-WIDE) show our method outperforms other existing
state-of-the-arts. In addition, we validate our method on a large multi-label
video classification dataset (YouTube-8M Segments) and the evaluation results
demonstrate the generalization capability of our method.

该研究提出了一种使用跨模态关注力和语义图嵌入的方法来进行多标签分类，该方法能够有效捕获标签之间的空间或时间依赖关系，并发现每个类的具有区分性特征的位置，实验结果表明该方法优于现有的最先进技术。