Democratic opinion-forming may be manipulated if newspapers' alignment to
political or economical orientation is ambiguous. Various methods have been
developed to better understand newspapers' positioning. Recently, the advent of
Large Language Models (LLM), and particularly the pre-trained LLM chatbots like
ChatGPT or Gemini, hold disruptive potential to assist researchers and citizens
alike. However, little is know on whether LLM assessment is trustworthy: do
single LLM agrees with experts' assessment, and do different LLMs answer
consistently with one another? In this paper, we address specifically the
second challenge. We compare how four widely employed LLMs rate the positioning
of newspapers, and compare if their answers align with one another. We observe
that this is not the case. Over a woldwide dataset, articles in newspapers are
positioned strikingly differently by single LLMs, hinting to inconsistent
training or excessive randomness in the algorithms. We thus raise a warning
when deciding which tools to use, and we call for better training and algorithm
development, to cover such significant gap in a highly sensitive matter for
democracy and societies worldwide. We also call for community engagement in
benchmark evaluation, through our open initiative navai.pro.

报纸定位存在明显差异问题，需加强 LLM 评估，改进算法以填补该民主国家敏感问题的重大空白，并促进社区参与。