The introduction of ChatGPT and the subsequent improvement of Large Language
Models (LLMs) have prompted more and more individuals to turn to the use of
ChatBots, both for information and assistance with decision-making. However,
the information the user is after is often not formulated by these ChatBots
objectively enough to be provided with a definite, globally accepted answer.
Controversial topics, such as "religion", "gender identity", "freedom of
speech", and "equality", among others, can be a source of conflict as partisan
or biased answers can reinforce preconceived notions or promote disinformation.
By exposing ChatGPT to such debatable questions, we aim to understand its level
of awareness and if existing models are subject to socio-political and/or
economic biases. We also aim to explore how AI-generated answers compare to
human ones. For exploring this, we use a dataset of a social media platform
created for the purpose of debating human-generated claims on polemic subjects
among users, dubbed Kialo.
Our results show that while previous versions of ChatGPT have had important
issues with controversial topics, more recent versions of ChatGPT
(gpt-3.5-turbo) are no longer manifesting significant explicit biases in
several knowledge areas. In particular, it is well-moderated regarding economic
aspects. However, it still maintains degrees of implicit libertarian leaning
toward right-winged ideals which suggest the need for increased moderation from
the socio-political point of view. In terms of domain knowledge on
controversial topics, with the exception of the "Philosophical" category,
ChatGPT is performing well in keeping up with the collective human level of
knowledge. Finally, we see that sources of Bing AI have slightly more tendency
to the center when compared to human answers. All the analyses we make are
generalizable to other types of biases and domains.

通过将 ChatGPT 暴露在具有争议性的问题上，我们旨在了解其意识水平，以及现有模型是否存在社会政治和 / 或经济偏见。同时，我们还旨在探讨人工智能生成的答案与人类答案的对比情况。通过使用社交媒体平台 Kialo 创建的数据集来进行探索。我们的研究结果表明，尽管 ChatGPT 的以前版本在争议性话题上存在重要问题，但最近的版本 (gpt-3.5-turbo) 在多个知识领域中不再表现出明显的显性偏见，特别是在经济方面进行了很好的调节。然而，它仍然保持着一定程度的对右倾意识形态的隐性倾向，这表明需要从社会政治的观点增加更多的调节。在争议话题的领域知识方面，除了 “哲学” 类别外，ChatGPT 在跟上人类集体知识水平方面表现良好。最后，我们发现与人类答案相比，Bing AI 的信息来源在倾向中立方面略有增加。我们的所有分析都具有普遍适用于其他类型的偏见和领域。