With recent advances in multi-modal foundation models, the previously
text-only large language models (LLM) have evolved to incorporate visual input,
opening up unprecedented opportunities for various applications in
visualization. Our work explores the utilization of the visual perception
ability of multi-modal LLMs to develop Autonomous Visualization Agents (AVAs)
that can interpret and accomplish user-defined visualization objectives through
natural language. We propose the first framework for the design of AVAs and
present several usage scenarios intended to demonstrate the general
applicability of the proposed paradigm. The addition of visual perception
allows AVAs to act as the virtual visualization assistant for domain experts
who may lack the knowledge or expertise in fine-tuning visualization outputs.
Our preliminary exploration and proof-of-concept agents suggest that this
approach can be widely applicable whenever the choices of appropriate
visualization parameters require the interpretation of previous visual output.
Feedback from unstructured interviews with experts in AI research, medical
visualization, and radiology has been incorporated, highlighting the
practicality and potential of AVAs. Our study indicates that AVAs represent a
general paradigm for designing intelligent visualization systems that can
achieve high-level visualization goals, which pave the way for developing
expert-level visualization agents in the future.

我们的研究探索多模式基础模型在可视化中的应用，利用多模式大语言模型 (LMM) 的视觉感知能力来开发自主可视化代理人 (AVAs)，它们可以通过自然语言解释和实现用户定义的可视化目标。我们提出了 AVAs 设计的第一个框架，并展示了几种使用场景，以展示该提议范例的普适性。这些 AVAs 作为虚拟可视化助手可以协助领域专家完成可视化输出的参数选择，即使他们缺乏调整可视化输出的知识或专业技能。我们的初步探索和概念验证代理人表明，这种方法在需要解释以往可视化输出以选择适当的可视化参数时可广泛应用。我们与人工智能研究、医学可视化和放射学等领域的专家进行了非结构化访谈，并结合了他们的反馈，突出了 AVAs 的实用性和潜力。我们的研究表明，AVAs 代表了一种设计智能化可视化系统的通用范式，可实现高级可视化目标，为未来发展专家级可视化代理人铺平了道路。