Neuro-Symbolic AI (NeSy) holds promise to ensure the safe deployment of AI
systems, as interpretable symbolic techniques provide formal behaviour
guarantees. The challenge is how to effectively integrate neural and symbolic
computation, to enable learning and reasoning from raw data. Existing pipelines
that train the neural and symbolic components sequentially require extensive
labelling, whereas end-to-end approaches are limited in terms of scalability,
due to the combinatorial explosion in the symbol grounding problem. In this
paper, we leverage the implicit knowledge within foundation models to enhance
the performance in NeSy tasks, whilst reducing the amount of data labelling and
manual engineering. We introduce a new architecture, called NeSyGPT, which
fine-tunes a vision-language foundation model to extract symbolic features from
raw data, before learning a highly expressive answer set program to solve a
downstream task. Our comprehensive evaluation demonstrates that NeSyGPT has
superior accuracy over various baselines, and can scale to complex NeSy tasks.
Finally, we highlight the effective use of a large language model to generate
the programmatic interface between the neural and symbolic components,
significantly reducing the amount of manual engineering required.

NeSyGPT 通过对视觉 - 语言基础模型进行微调，从原始数据中提取符号特征，并学习一个高度表达的回答集程序来解决下游任务，以提高 NeSy 任务的性能和减少数据标记和手动工程工作量。