With the emergence of large language models (LLMs) and vision foundation
models, how to combine the intelligence and capacity of these open-sourced or
API-available models to achieve open-world visual perception remains an open
question. In this paper, we introduce VisionGPT to consolidate and automate the
integration of state-of-the-art foundation models, thereby facilitating
vision-language understanding and the development of vision-oriented AI.
VisionGPT builds upon a generalized multimodal framework that distinguishes
itself through three key features: (1) utilizing LLMs (e.g., LLaMA-2) as the
pivot to break down users' requests into detailed action proposals to call
suitable foundation models; (2) integrating multi-source outputs from
foundation models automatically and generating comprehensive responses for
users; (3) adaptable to a wide range of applications such as text-conditioned
image understanding/generation/editing and visual question answering. This
paper outlines the architecture and capabilities of VisionGPT, demonstrating
its potential to revolutionize the field of computer vision through enhanced
efficiency, versatility, and generalization, and performance. Our code and
models will be made publicly available. Keywords: VisionGPT, Open-world visual
perception, Vision-language understanding, Large language model, and Foundation
model

介绍 VisionGPT，通过整合和自动化最先进的基础模型，促进视觉语言理解和视觉导向人工智能的发展，具有高效性、通用性和性能表现的潜力。