This study evaluates the ability of Large Language Model (LLM)-based
Subpopulation Representative Models (SRMs) to generalize from empirical data,
utilizing in-context learning with data from the 2016 and 2020 American
National Election Studies. We explore generalization across response variables
and demographic subgroups. While conditioning with empirical data improves
performance on the whole, the benefit of in-context learning varies
considerably across demographics, sometimes hurting performance for one
demographic while helping performance for others. The inequitable benefits of
in-context learning for SRM present a challenge for practitioners implementing
SRMs, and for decision-makers who might come to rely on them. Our work
highlights a need for fine-grained benchmarks captured from diverse
subpopulations that test not only fidelity but generalization.

该研究评估了基于大型语言模型（LLM）的子群代表模型（SRM）利用 2016 年和 2020 年美国全国选举研究的实证数据进行从经验数据中概括的能力，研究泛化能力在不同的响应变量和人口统计学子群之间的差异，并指出在语境学习下的表现不同对各人口统计学子群的效果也不同，有时改善了某个人口统计学子群的表现却损害了其他人口统计学子群的表现，SRM 在语境学习下的不均衡效果对于实施 SRM 的从业者以及依赖 SRM 的决策者提出了挑战，我们的研究突显了从各个不同子群中获得精细化的基准测试数据的需求，这些测试不仅要考虑保真度还要考虑泛化能力。