In this paper, we propose an autonomous information seeking visual question
answering framework, AVIS. Our method leverages a Large Language Model (LLM) to
dynamically strategize the utilization of external tools and to investigate
their outputs, thereby acquiring the indispensable knowledge needed to provide
answers to the posed questions. Responding to visual questions that necessitate
external knowledge, such as "What event is commemorated by the building
depicted in this image?", is a complex task. This task presents a combinatorial
search space that demands a sequence of actions, including invoking APIs,
analyzing their responses, and making informed decisions. We conduct a user
study to collect a variety of instances of human decision-making when faced
with this task. This data is then used to design a system comprised of three
components: an LLM-powered planner that dynamically determines which tool to
use next, an LLM-powered reasoner that analyzes and extracts key information
from the tool outputs, and a working memory component that retains the acquired
information throughout the process. The collected user behavior serves as a
guide for our system in two key ways. First, we create a transition graph by
analyzing the sequence of decisions made by users. This graph delineates
distinct states and confines the set of actions available at each state.
Second, we use examples of user decision-making to provide our LLM-powered
planner and reasoner with relevant contextual instances, enhancing their
capacity to make informed decisions. We show that AVIS achieves
state-of-the-art results on knowledge-intensive visual question answering
benchmarks such as Infoseek and OK-VQA.

本文提出了一种基于大型语言模型的自主信息查找视觉问答框架 (AVIS)，包括三个组件：动态决定下一个工具的 LLM 规划器、分析和提取工具输出关键信息的 LLM 推理器，以及在整个过程中保留所获取信息的工作记忆组件。通过用户行为作为向导，该系统通过分析用户决策顺序创建转换图，并利用用户决策制定决策策略，从而实现了在 Infoseek 和 OK-VQA 等知识密集型视觉问答基准测试中取得了最新的成果。

基于大型语言模型的自主视觉信息获取

AVIS: Autonomous Visual Information Seeking with Large Language Models

We propose a generative machine comprehension model that learns jointly to
ask and answer questions based on documents. The proposed model uses a
sequence-to-sequence framework that encodes the document and generates a
question (answer) given an answer (question). Significant improvement in model
performance is observed empirically on the SQuAD corpus, confirming our
hypothesis that the model benefits from jointly learning to perform both tasks.
We believe the joint model's novelty offers a new perspective on machine
comprehension beyond architectural engineering, and serves as a first step
towards autonomous information seeking.

本文提出一种生成式机器理解模型，用于同时学习根据文档提问和回答问题，采用序列到序列框架编码文档并生成问题，模型在 SQuAD 语料库上获得显著性能提升，实验证明，模型在同时学习两个任务方面受益。我们认为这种联合模型的新颖性提供了一个超越架构工程的机器理解视角，是实现自主信息寻找的第一步。