While neural symbolic methods demonstrate impressive performance in visual
question answering on synthetic images, their performance suffers on real
images. We identify that the long-tail distribution of visual concepts and
unequal importance of reasoning steps in real data are the two key obstacles
that limit the models' real-world potentials. To address these challenges, we
propose a new paradigm, Calibrating Concepts and Operations (CCO), which
enables neural symbolic models to capture underlying data characteristics and
to reason with hierarchical importance. Specifically, we introduce an executor
with learnable concept embedding magnitudes for handling distribution
imbalance, and an operation calibrator for highlighting important operations
and suppressing redundant ones. Our experiments show CCO substantially boosts
the performance of neural symbolic methods on real images. By evaluating models
on the real world dataset GQA, CCO helps the neural symbolic method NSCL
outperforms its vanilla counterpart by 9.1% (from 47.0% to 56.1%); this result
also largely reduces the performance gap between symbolic and non-symbolic
methods. Additionally, we create a perturbed test set for better understanding
and analyzing model performance on real images. Code is available at
this https URL .

提出了 Calibrating Concepts and Operations（CCO）方法，旨在通过学习概念嵌入矢量和操作校准器解决真实图像数据中长尾概念分布和推断步骤不平等重要性的问题，并在真实世界数据集 GQA 上大幅提升神经符号方法的性能。