Large vision-language models (LVLMs) are ignorant of the up-to-date
knowledge, such as LLaVA series, because they cannot be updated frequently due
to the large amount of resources required, and therefore fail in many cases.
For example, if a LVLM was released on January 2024, and it wouldn't know the
detailed plot of the new movie Dune 2, which wasn't released until February
2024. To solve the problem, a promising solution is to provide LVLMs with
up-to-date knowledge via internet search during inference, i.e.,
internet-augmented generation (IAG), which is already integrated in some
closed-source commercial LVLMs such as GPT-4V. However, the specific mechanics
underpinning them remain a mystery. In this paper, we propose a plug-and-play
framework, for augmenting existing LVLMs in handling visual question answering
(VQA) about up-to-date knowledge, dubbed UDKAG. A hierarchical filtering model
is trained to effectively and efficiently find the most helpful content from
the websites returned by a search engine to prompt LVLMs with up-to-date
knowledge. To train the model and evaluate our framework's performance, we
propose a pipeline to automatically generate news-related VQA samples to
construct a dataset, dubbed UDK-VQA. A multi-model voting mechanism is
introduced to label the usefulness of website/content for VQA samples to
construct the training set. Experimental results demonstrate the effectiveness
of our framework, outperforming GPT-4V by about 25% in accuracy.

提出了一种名为 UDKAG 的框架，通过互联网搜索来为 Large vision-language models 提供最新的知识，以应对视觉问答。使用层级过滤模型从搜索引擎返回的网站中有效地找到最有帮助的内容。实验结果表明，该框架的效果比 GPT-4V 高出约 25%。