Recent advancements in visiolinguistic (VL) learning have allowed the
development of multiple models and techniques that offer several impressive
implementations, able to currently resolve a variety of tasks that require the
collaboration of vision and language. Current datasets used for VL pre-training
only contain a limited amount of visual and linguistic knowledge, thus
significantly limiting the generalization capabilities of many VL models.
External knowledge sources such as knowledge graphs (KGs) and Large Language
Models (LLMs) are able to cover such generalization gaps by filling in missing
knowledge, resulting in the emergence of hybrid architectures. In the current
survey, we analyze tasks that have benefited from such hybrid approaches.
Moreover, we categorize existing knowledge sources and types, proceeding to
discussion regarding the KG vs LLM dilemma and its potential impact to future
hybrid approaches.

研究了基于视觉和语言协同的任务，考虑利用知识图谱和大型语言模型等外部知识来源填补现有 VL 预训练数据集中存在的知识缺失问题，提出了基于混合架构的解决方案，并总结了知识图谱与大型语言模型潜在对未来混合模型的影响。

视觉语言学习中知识的贡献：任务和挑战调查

The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges

Multimodal learning has been a field of increasing interest, aiming to
combine various modalities in a single joint representation. Especially in the
area of visiolinguistic (VL) learning multiple models and techniques have been
developed, targeting a variety of tasks that involve images and text. VL models
have reached unprecedented performances by extending the idea of Transformers,
so that both modalities can learn from each other. Massive pre-training
procedures enable VL models to acquire a certain level of real-world
understanding, although many gaps can be identified: the limited comprehension
of commonsense, factual, temporal and other everyday knowledge aspects
questions the extendability of VL tasks. Knowledge graphs and other knowledge
sources can fill those gaps by explicitly providing missing information,
unlocking novel capabilities of VL models. In the same time, knowledge graphs
enhance explainability, fairness and validity of decision making, issues of
outermost importance for such complex implementations. The current survey aims
to unify the fields of VL representation learning and knowledge graphs, and
provides a taxonomy and analysis of knowledge-enhanced VL models.

该研究调查了多模态学习中的视觉语言学习及其与知识图谱的结合，提出了一种基于知识图谱的视觉语言学习模型以解决一般性知识方面的问题。