Large Language Models (LLMs) have gained significant popularity for their
application in various everyday tasks such as text generation, summarization,
and information retrieval. As the widespread adoption of LLMs continues to
surge, it becomes increasingly crucial to ensure that these models yield
responses that are politically impartial, with the aim of preventing
information bubbles, upholding fairness in representation, and mitigating
confirmation bias. In this paper, we propose a quantitative framework and
pipeline designed to systematically investigate the political orientation of
LLMs. Our investigation delves into the political alignment of LLMs across a
spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues.
Across topics, the results indicate that LLMs exhibit a tendency to provide
responses that closely align with liberal or left-leaning perspectives rather
than conservative or right-leaning ones when user queries include details
pertaining to occupation, race, or political affiliation. The findings
presented in this study not only reaffirm earlier observations regarding the
left-leaning characteristics of LLMs but also surface particular attributes,
such as occupation, that are particularly susceptible to such inclinations even
when directly steered towards conservatism. As a recommendation to avoid these
models providing politicised responses, users should be mindful when crafting
queries, and exercise caution in selecting neutral prompt language.

利用定量框架和流程系统地调查大型语言模型的政治取向，研究结果显示在八个极化话题中，当用户查询涉及职业、种族或政治取向时，大型语言模型倾向于提供与自由派或左倾观点更为接近的回答，而不是保守派或右倾观点。为了避免这些模型提供政治化的回答，用户在构建查询时应谨慎，并选择中立的提示语言。

大型语言模型的政治立场调查

Whose Side Are You On? Investigating the Political Stance of Large  Language Models

The surprisingly likely criterion in the seminal work of Prelec (the Bayesian
Truth Serum) guarantees truthfulness in a game-theoretic multi-agent setting,
by rewarding rational agents to maximise the expected information gain with
their answers w.r.t. their probabilistic beliefs. We investigate the relevance
of a similar criterion for responses of LLMs. We hypothesize that if the
surprisingly likely criterion works in LLMs, under certain conditions, the
responses that maximize the reward under this criterion should be more accurate
than the responses that only maximize the posterior probability. Using
benchmarks including the TruthfulQA benchmark and using openly available LLMs:
GPT-2 and LLaMA-2, we show that the method indeed improves the accuracy
significantly (for example, upto 24 percentage points aggregate improvement on
TruthfulQA and upto 70 percentage points improvement on individual categories
of questions).

在这篇研究中，我们调查了类似于 Bayesian Truth Serum 的标准对 LLMs 的回答的相关性，并假设在一定条件下，根据这一标准最大化奖励的回答应该比仅最大化后验概率的回答更准确。通过使用包括 TruthfulQA 基准和开放可用的 LLMs（如 GPT-2 和 LLaMA-2）的基准实验证明该方法确实显著提高了准确性（例如，在 TruthfulQA 上的总体改进高达 24 个百分点，在各个问题类别上的改进高达 70 个百分点）。