Today, using Large-scale generative Language Models (LLMs) it is possible to
simulate free responses to interview questions like those traditionally
analyzed using qualitative research methods. Qualitative methodology
encompasses a broad family of techniques involving manual analysis of
open-ended interviews or conversations conducted freely in natural language.
Here we consider whether artificial "silicon participants" generated by LLMs
may be productively studied using qualitative methods aiming to produce
insights that could generalize to real human populations. The key concept in
our analysis is algorithmic fidelity, a term introduced by Argyle et al. (2023)
capturing the degree to which LLM-generated outputs mirror human
sub-populations' beliefs and attitudes. By definition, high algorithmic
fidelity suggests latent beliefs elicited from LLMs may generalize to real
humans, whereas low algorithmic fidelity renders such research invalid. Here we
used an LLM to generate interviews with silicon participants matching specific
demographic characteristics one-for-one with a set of human participants. Using
framework-based qualitative analysis, we showed the key themes obtained from
both human and silicon participants were strikingly similar. However, when we
analyzed the structure and tone of the interviews we found even more striking
differences. We also found evidence of the hyper-accuracy distortion described
by Aher et al. (2023). We conclude that the LLM we tested (GPT-3.5) does not
have sufficient algorithmic fidelity to expect research on it to generalize to
human populations. However, the rapid pace of LLM research makes it plausible
this could change in the future. Thus we stress the need to establish epistemic
norms now around how to assess validity of LLM-based qualitative research,
especially concerning the need to ensure representation of heterogeneous lived
experiences.

使用大规模生成性语言模型（LLMs）可以模拟自由回答面试问题，这种方法可以代替传统的定性研究方法；研究中提出了算法忠实度的概念，通过框架化定性分析发现人工和模型生成的参与者得到的主题相似，但在访谈的结构和语气上存在显著差异；LLMs 目前的算法忠实度不足以推广到人类群体，但未来可能会有改变；因此，有必要建立关于如何评估基于 LLMs 的定性研究的认知规范，特别是关于确保包含多样化生活经历的代表性。

基于框架的大型语言模型自由回答的定性分析：算法保真度

Framework-Based Qualitative Analysis of Free Responses of Large Language  Models: Algorithmic Fidelity

The recent advance of self-supervised learning associated with the
Transformer architecture enables natural language processing (NLP) to exhibit
extremely low perplexity. Such powerful models demand ever-increasing model
size and, thus, large amounts of computations and memory footprints. In this
paper, we propose an efficient inference framework for large-scale generative
language models. As the key to reducing model size, we quantize weights by a
non-uniform quantization method. Then, quantized matrix multiplications are
accelerated by our proposed kernel, called nuQmm, which allows a wide trade-off
between compression ratio and accuracy. Our proposed nuQmm reduces the latency
of not only each GPU but also the entire inference of large LMs because a high
compression ratio (by low-bit quantization) mitigates the minimum required
number of GPUs. Assuming 2-bit quantization, we demonstrate that nuQmm can
reduce latency to generate each token for OPT-175B (that requires 8 GPUs
without nuQmm) by 47.3% using 8 GPUs or by 23.2% using only 2 GPUs.

该论文提出了一个有效的推理框架用于大规模生成式语言模型，采用自我监督学习和 Transformer 架构的最新进展实现了低困惑度，通过非均匀量化和 nuQmm 的加速矩阵乘法，实现了模型大小的减小，并减少了大型 LM 的推理延迟