We investigate to which extent open large language models (LLMs) can generate
coherent and relevant text from structured data. To prevent bias from
benchmarks leaked into LLM training data, we collect Quintd-1: an ad-hoc
benchmark for five data-to-text (D2T) generation tasks, consisting of
structured data records in standard formats gathered from public APIs. We
leverage reference-free evaluation metrics and LLMs' in-context learning
capabilities, allowing us to test the models with no human-written references.
Our evaluation focuses on annotating semantic accuracy errors on token-level,
combining human annotators and a metric based on GPT-4. Our systematic
examination of the models' behavior across domains and tasks suggests that
state-of-the-art open LLMs with 7B parameters can generate fluent and coherent
text from various standard data formats in zero-shot settings. However, we also
show that semantic accuracy of the outputs remains a major issue: on our
benchmark, 80% of outputs of open LLMs contain a semantic error according to
human annotators (91% according to GPT-4). Our code, data, and model outputs
are available at this https URL

探究了开放式大型语言模型（LLMs）从结构化数据生成连贯和相关文本的程度，通过使用 Quintd-1 数据集，以及基于语义准确性错误的评估方法，发现开放式 LLMs 可以在零样本情况下从各种标准数据格式生成流畅、一致的文本，但语义准确性仍然是一个主要问题。

超越基于参考度量的方法：分析数据到文本生成领域的开放式 LLM 行为

Beyond Reference-Based Metrics: Analyzing Behaviors of Open LLMs on  Data-to-Text Generation

The emergence of accurate open large language models (LLMs) has led to a race
towards quantization techniques for such models enabling execution on end-user
devices. In this paper, we revisit the problem of "extreme" LLM
compression--defined as targeting extremely low bit counts, such as 2 to 3 bits
per parameter, from the point of view of classic methods in Multi-Codebook
Quantization (MCQ). Our work builds on top of Additive Quantization, a classic
algorithm from the MCQ family, and adapts it to the quantization of language
models. The resulting algorithm advances the state-of-the-art in LLM
compression, outperforming all recently-proposed techniques in terms of
accuracy at a given compression budget. For instance, when compressing Llama 2
models to 2 bits per parameter, our algorithm quantizes the 7B model to 6.93
perplexity (a 1.29 improvement relative to the best prior work, and 1.81 points
from FP16), the 13B model to 5.70 perplexity (a .36 improvement) and the 70B
model to 3.94 perplexity (a .22 improvement) on WikiText2. We release our
implementation of Additive Quantization for Language Models AQLM as a baseline
to facilitate future research in LLM quantization.

该研究中，我们将经典的多码本量化方法应用于语言模型的压缩，实现了对开放式大规模语言模型的极端压缩，提高了在给定压缩预算下的准确性。