We study the problem of automatically building hypernym taxonomies from
textual and visual data. Previous works in taxonomy induction generally ignore
the increasingly prominent visual data, which encode important perceptual
semantics. Instead, we propose a probabilistic model for taxonomy induction by
jointly leveraging text and images. To avoid hand-crafted feature engineering,
we design end-to-end features based on distributed representations of images
and words. The model is discriminatively trained given a small set of existing
ontologies and is capable of building full taxonomies from scratch for a
collection of unseen conceptual label items with associated images. We evaluate
our model and features on the WordNet hierarchies, where our system outperforms
previous approaches by a large gap.

本文提出了一种基于文本和图像的概率模型，通过端到端特征设计实现自动构建上位词分类法。通过小型本体数据的判别训练，该模型可以从头开始为具有关联图像的未知概念标签构建完整的分类法。在 WordNet 层次结构的评估中，本系统的表现优于以往方法。