The complexity of the visual world creates significant challenges for
comprehensive visual understanding. In spite of recent successes in visual
recognition, today's vision systems would still struggle to deal with visual
queries that require a deeper reasoning. We propose a knowledge base (KB)
framework to handle an assortment of visual queries, without the need to train
new classifiers for new tasks. Building such a large-scale multimodal KB
presents a major challenge of scalability. We cast a large-scale MRF into a KB
representation, incorporating visual, textual and structured data, as well as
their diverse relations. We introduce a scalable knowledge base construction
system that is capable of building a KB with half billion variables and
millions of parameters in a few hours. Our system achieves competitive results
compared to purpose-built models on standard recognition and retrieval tasks,
while exhibiting greater flexibility in answering richer visual queries.

本研究提出了一种知识库框架，通过构建一个大规模的多模态知识库来回答各种视觉查询，同时保持灵活性和可扩展性。研究表明所提系统能够取得有竞争力的结果，并能够应对更丰富的视觉查询。