Leveraging large language models (LLMs) to integrate off-the-shelf tools
(e.g., visual models and image processing functions) is a promising research
direction to build powerful visual assistants for solving diverse visual tasks.
However, the learning capability is rarely explored in existing methods, as
they freeze the used tools after deployment, thereby limiting the
generalization to new environments requiring specific knowledge. In this paper,
we propose CLOVA, a Closed-LOop Visual Assistant to address this limitation,
which encompasses inference, reflection, and learning phases in a closed-loop
framework. During inference, LLMs generate programs and execute corresponding
tools to accomplish given tasks. The reflection phase introduces a multimodal
global-local reflection scheme to analyze whether and which tool needs to be
updated based on environmental feedback. Lastly, the learning phase uses three
flexible manners to collect training data in real-time and introduces a novel
prompt tuning scheme to update the tools, enabling CLOVA to efficiently learn
specific knowledge for new environments without human involvement. Experiments
show that CLOVA outperforms tool-usage methods by 5% in visual question
answering and multiple-image reasoning tasks, by 10% in knowledge tagging
tasks, and by 20% in image editing tasks, highlighting the significance of the
learning capability for general visual assistants.

本文介绍了一种闭环视觉助手 CLOVA，利用大型语言模型（LLMs）将现成工具集成在一起，通过推理、反馈和学习三个阶段来解决视觉助手在新环境中推广使用的能力有限的问题。实验结果显示，CLOVA 在视觉问题回答、知识标注和图像编辑任务方面的性能均超过其他方法。