Translating natural language to visualization (NL2VIS) has shown great
promise for visual data analysis, but it remains a challenging task that
requires multiple low-level implementations, such as natural language
processing and visualization design. Recent advancements in pre-trained large
language models (LLMs) are opening new avenues for generating visualizations
from natural language. However, the lack of a comprehensive and reliable
benchmark hinders our understanding of LLMs' capabilities in visualization
generation. In this paper, we address this gap by proposing a new NL2VIS
benchmark called VisEval. Firstly, we introduce a high-quality and large-scale
dataset. This dataset includes 2,524 representative queries covering 146
databases, paired with accurately labeled ground truths. Secondly, we advocate
for a comprehensive automated evaluation methodology covering multiple
dimensions, including validity, legality, and readability. By systematically
scanning for potential issues with a number of heterogeneous checkers, VisEval
provides reliable and trustworthy evaluation outcomes. We run VisEval on a
series of state-of-the-art LLMs. Our evaluation reveals prevalent challenges
and delivers essential insights for future advancements.

提出了一种名为 VisEval 的新的 NL2VIS 基准，该基准通过引入高质量大规模数据集并倡导综合自动化评估方法，揭示了目前研究中普遍存在的挑战并为未来的进一步发展提供了重要的见解。

大语言模型时代的数据可视化基准测试：VisEval

VisEval: A Benchmark for Data Visualization in the Era of Large Language  Models

NL2VIS (natural language to visualization) is a promising and recent research
area that involves interpreting natural language queries and translating them
into visualizations that accurately represent the underlying data. As we
navigate the era of big data, NL2VIS holds considerable application potential
since it greatly facilitates data exploration by non-expert users. Following
the increasingly widespread usage of generative AI in NL2VIS applications, in
this paper we present V-RECS, the first LLM-based Visual Recommender augmented
with explanations(E), captioning(C), and suggestions(S) for further data
exploration. V-RECS' visualization narratives facilitate both response
verification and data exploration by non-expert users. Furthermore, our
proposed solution mitigates computational, controllability, and cost issues
associated with using powerful LLMs by leveraging a methodology to effectively
fine-tune small models. To generate insightful visualization narratives, we use
Chain-of-Thoughts (CoT), a prompt engineering technique to help LLM identify
and generate the logical steps to produce a correct answer. Since CoT is
reported to perform poorly with small LLMs, we adopted a strategy in which a
large LLM (GPT-4), acting as a Teacher, generates CoT-based instructions to
fine-tune a small model, Llama-2-7B, which plays the role of a Student.
Extensive experiments-based on a framework for the quantitative evaluation of
AI-based visualizations and on manual assessment by a group of
participants-show that V-RECS achieves performance scores comparable to GPT-4,
at a much lower cost. The efficacy of the V-RECS teacher-student paradigm is
also demonstrated by the fact that the un-tuned Llama fails to perform the task
in the vast majority of test cases. We release V-RECS for the visualization
community to assist visualization designers throughout the entire visualization
generation process.

NL2VIS 研究领域涉及将自然语言查询解释并将其转化为准确代表底层数据的可视化，为非专家用户大大便利了数据探索；本文提出了 V-RECS，首个具有解释、标题和进一步数据探索建议的基于 LLM 的视觉推荐系统，通过细化小型模型的方法解决了强大 LLMs 的计算、可控性和成本问题，使用 Chain-of-Thoughts（CoT）技术来生成有洞察力的可视化叙述来辅助 LLM 的指导，V-RECS 在性能上与 GPT-4 相当，但成本更低，该研究通过发布 V-RECS 帮助可视化设计师整个可视化生成过程。

V-RECS, 一种低成本的 LLM4VIS 推荐器，附有解释、字幕和建议

V-RECS, a Low-Cost LLM4VIS Recommender with Explanations, Captioning and  Suggestions

The Natural Language to Visualization (NL2Vis) task aims to transform
natural-language descriptions into visual representations for a grounded table,
enabling users to gain insights from vast amounts of data. Recently, many deep
learning-based approaches have been developed for NL2Vis. Despite the
considerable efforts made by these approaches, challenges persist in
visualizing data sourced from unseen databases or spanning multiple tables.
Taking inspiration from the remarkable generation capabilities of Large
Language Models (LLMs), this paper conducts an empirical study to evaluate
their potential in generating visualizations, and explore the effectiveness of
in-context learning prompts for enhancing this task. In particular, we first
explore the ways of transforming structured tabular data into sequential text
prompts, as to feed them into LLMs and analyze which table content contributes
most to the NL2Vis. Our findings suggest that transforming structured tabular
data into programs is effective, and it is essential to consider the table
schema when formulating prompts. Furthermore, we evaluate two types of LLMs:
finetuned models (e.g., T5-Small) and inference-only models (e.g., GPT-3.5),
against state-of-the-art methods, using the NL2Vis benchmarks (i.e., nvBench).
The experimental results reveal that LLMs outperform baselines, with
inference-only models consistently exhibiting performance improvements, at
times even surpassing fine-tuned models when provided with certain few-shot
demonstrations through in-context learning. Finally, we analyze when the LLMs
fail in NL2Vis, and propose to iteratively update the results using strategies
such as chain-of-thought, role-playing, and code-interpreter. The experimental
results confirm the efficacy of iterative updates and hold great potential for
future study.

使用大型语言模型进行自然语言到可视化的转换任务，在分析如何将结构化表格数据转化为 LLM 所需的顺序文本提示时，发现将结构化表格数据转化为程序是有效的，并且在制定提示时考虑表格模式是必要的。同时，通过对比实验发现，LLM 在 NL2Vis 任务中优于基线方法，并且推理模型在提供少量示范的情况下通过上下文学习能够进一步改善性能，时而甚至超过微调模型。最后，分析 LLM 在 NL2Vis 任务中失败的情况，并提出了循环更新的策略，通过链式思维、角色扮演和代码解释等方法来迭代更新结果，实验证实了这种迭代更新的有效性，具有广阔的未来研究潜力。