We propose Neuro-Symbolic Visual Dialog (NSVD) -the first method to combine
deep learning and symbolic program execution for multi-round visually-grounded
reasoning. NSVD significantly outperforms existing purely-connectionist methods
on two key challenges inherent to visual dialog: long-distance co-reference
resolution as well as vanishing question-answering performance. We demonstrate
the latter by proposing a more realistic and stricter evaluation scheme in
which we use predicted answers for the full dialog history when calculating
accuracy. We describe two variants of our model and show that using this new
scheme, our best model achieves an accuracy of 99.72% on CLEVR-Dialog -a
relative improvement of more than 10% over the state of the art while only
requiring a fraction of training data. Moreover, we demonstrate that our
neuro-symbolic models have a higher mean first failure round, are more robust
against incomplete dialog histories, and generalise better not only to dialogs
that are up to three times longer than those seen during training but also to
unseen question types and scenes.

提出神经符号视觉对话（NSVD）方法，将深度学习和符号程序执行结合起来，用于多轮基于视觉的推理，并取得了显著的性能提升。在使用更严格的评估方案时，我们的最佳模型在 CLEVR-Dialog 数据集上实现了 99.72％的准确度，在总体上比现有技术提高了 10％以上。我们的结果表明，神经符号模型的平均故障轮数更高，对不完整的对话历史记录更稳健，并在长度高达训练期间三倍的对话框架以及未见过的问题类型和场景上表现出更好的泛化能力。

神经符号视觉对话

Neuro-Symbolic Visual Dialog

We marry two powerful ideas: deep representation learning for visual
recognition and language understanding, and symbolic program execution for
reasoning. Our neural-symbolic visual question answering (NS-VQA) system first
recovers a structural scene representation from the image and a program trace
from the question. It then executes the program on the scene representation to
obtain an answer. Incorporating symbolic structure as prior knowledge offers
three unique advantages. First, executing programs on a symbolic space is more
robust to long program traces; our model can solve complex reasoning tasks
better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model
is more data- and memory-efficient: it performs well after learning on a small
number of training data; it can also encode an image into a compact
representation, requiring less storage than existing methods for offline
question answering. Third, symbolic program execution offers full transparency
to the reasoning process; we are thus able to interpret and diagnose each
execution step.

该论文介绍了神经符号视觉问答系统，结合深度表示学习和符号程序执行，具有更好的数据和内存效率以及对推理过程的全透明度。