Humans have the innate capability to answer diverse questions, which is
rooted in the natural ability to correlate different concepts based on their
semantic relationships and decompose difficult problems into sub-tasks. On the
contrary, existing visual reasoning methods assume training samples that
capture every possible object and reasoning problem, and rely on black-boxed
models that commonly exploit statistical priors. They have yet to develop the
capability to address novel objects or spurious biases in real-world scenarios,
and also fall short of interpreting the rationales behind their decisions.
Inspired by humans' reasoning of the visual world, we tackle the aforementioned
challenges from a compositional perspective, and propose an integral framework
consisting of a principled object factorization method and a novel neural
module network. Our factorization method decomposes objects based on their key
characteristics, and automatically derives prototypes that represent a wide
range of objects. With these prototypes encoding important semantics, the
proposed network then correlates objects by measuring their similarity on a
common semantic space and makes decisions with a compositional reasoning
process. It is capable of answering questions with diverse objects regardless
of their availability during training, and overcoming the issues of biased
question-answer distributions. In addition to the enhanced generalizability,
our framework also provides an interpretable interface for understanding the
decision-making process of models. Our code is available at
this https URL

本文提出了一种基于物体分解和语义关系的视觉推理框架，包括物体因子分解方法和神经模块网络，具有较好的推理能力和泛化性能，并且提供了可解释的决策过程界面。

分治策略：基于对象分解和组合推理的问题解答

Divide and Conquer: Answering Questions with Object Factorization and  Compositional Reasoning

Many vision and language tasks require commonsense reasoning beyond
data-driven image and natural language processing. Here we adopt Visual
Question Answering (VQA) as an example task, where a system is expected to
answer a question in natural language about an image. Current state-of-the-art
systems attempted to solve the task using deep neural architectures and
achieved promising performance. However, the resulting systems are generally
opaque and they struggle in understanding questions for which extra knowledge
is required. In this paper, we present an explicit reasoning layer on top of a
set of penultimate neural network based systems. The reasoning layer enables
reasoning and answering questions where additional knowledge is required, and
at the same time provides an interpretable interface to the end users.
Specifically, the reasoning layer adopts a Probabilistic Soft Logic (PSL) based
engine to reason over a basket of inputs: visual relations, the semantic parse
of the question, and background ontological knowledge from word2vec and
ConceptNet. Experimental analysis of the answers and the key evidential
predicates generated on the VQA dataset validate our approach.

该论文提出了一种在视觉问答任务中使用通感推理的方法，该方法可以通过显式推理层来解决一些需要额外知识的问题，并提供可解释性的界面。该推理层采用了基于概率的软逻辑引擎，并运用了一篮子输入来进行推理。实验证实了该方法的有效性。