Discovering automatically the semantic structure of tagged visual data (e.g. web videos and images) is important for visual data analysis and interpretation, enabling the machine intelligence for effectively processing the fast-growing amount of multi-media data. However, this is non-trivial due to the need for jointly learning underlying correlations between heterogeneous visual and tag data. The task is made more challenging by inherently sparse and incomplete tags. In this work, we develop a method for modelling the inherent visual data concept structures based on a novel Hierarchical-Multi-Label Random Forest model capable of correlating structured visual and tag information so as to more accurately interpret the visual semantics, e.g. disclosing meaningful visual groups with similar high-level concepts, and recovering missing tags for individual visual data samples. Specifically, our model exploits hierarchically structured tags of different semantic abstractness and multiple tag statistical correlations in addition to modelling visual and tag interactions. As a result, our model is able to discover more accurate semantic correlation between textual tags and visual features, and finally providing favourable visual semantics interpretation even with highly sparse and incomplete tags. We demonstrate the advantages of our proposed approach in two fundamental applications, visual data clustering and missing tag completion, on benchmarking video (i.e. TRECVID MED 2011) and image (i.e. NUS-WIDE) datasets.

该研究提出一种基于层次多标签随机森林模型的方法，能够将视觉数据的概念结构与标签信息进行相关联，并能够准确地解释视觉语义，例如揭示具有类似高级概念的有意义的视觉组，并为单个视觉数据样本恢复缺失标签。此方法能够在高度稀疏和不完整的标签情况下，发现更准确的文本标签和视觉特征之间的语义相关性，并在基准视频和图像数据集上展示了其应用于视觉数据聚类和缺失标记完成两个基本应用的优点。

利用稀疏不完整标记发现视觉概念结构