How to build a machine learning method that can continuously gain structured visual knowledge by learning structured facts? Our goal in this paper is to address this question by proposing a problem setting, where training data comes as structured facts in images with different types including (1) objects(e.g., < boy >), (2) attributes (e.g., < boy,tall >), (3) actions (e.g., < boy, playing >), (4) interactions (e.g., < boy, riding, a horse >). Each structured fact has a semantic language view (e.g., < boy, playing >) and a visual view (an image with this fact). A human is able to efficiently gain visual knowledge by learning facts in a never ending process, and as we believe in a structured way (e.g., understanding "playing" is the action part of < boy, playing >, and hence can generalize to recognize < girl, playing > if just learn < girl > additionally). Inspired by human visual perception, we propose a model that is (1) able to learn a representation, we name as wild-card, which covers different types of structured facts, (2) could flexibly get fed with structured fact language-visual view pairs in a never ending way to gain more structured knowledge, (3) could generalize to unseen facts, and (4) allows retrieval of both the fact language view given the visual view (i.e., image) and vice versa. We also propose a novel method to generate hundreds of thousands of structured fact pairs from image caption data, which are necessary to train our model and can be useful for other applications.

本研究提出了在图像中可扩展和统一地理解事实的方案，通过学习有结构的视觉事实，实现了一种均一且具有通用性的视觉理解，并在拓展了结构化事实的多个数据集以及超过202,000个事实和814,000个图像的大规模数据集中应用了所研究的方法。

Sherlock: 图像中可扩展的事实学习