This article presents a comparative analysis of the ability of two large
language model (LLM)-based chatbots, ChatGPT and Bing Chat, recently rebranded
to Microsoft Copilot, to detect veracity of political information. We use AI
auditing methodology to investigate how chatbots evaluate true, false, and
borderline statements on five topics: COVID-19, Russian aggression against
Ukraine, the Holocaust, climate change, and LGBTQ+ related debates. We compare
how the chatbots perform in high- and low-resource languages by using prompts
in English, Russian, and Ukrainian. Furthermore, we explore the ability of
chatbots to evaluate statements according to political communication concepts
of disinformation, misinformation, and conspiracy theory, using
definition-oriented prompts. We also systematically test how such evaluations
are influenced by source bias which we model by attributing specific claims to
various political and social actors. The results show high performance of
ChatGPT for the baseline veracity evaluation task, with 72 percent of the cases
evaluated correctly on average across languages without pre-training. Bing Chat
performed worse with a 67 percent accuracy. We observe significant disparities
in how chatbots evaluate prompts in high- and low-resource languages and how
they adapt their evaluations to political communication concepts with ChatGPT
providing more nuanced outputs than Bing Chat. Finally, we find that for some
veracity detection-related tasks, the performance of chatbots varied depending
on the topic of the statement or the source to which it is attributed. These
findings highlight the potential of LLM-based chatbots in tackling different
forms of false information in online environments, but also points to the
substantial variation in terms of how such potential is realized due to
specific factors, such as language of the prompt or the topic.

本研究比较分析了两个大型语言模型（LLM）聊天机器人 ——ChatGPT 和 Bing Chat（现在已更名为 Microsoft Copilot）在检测政治信息真实性方面的能力。通过使用人工智能审计方法，我们在 COVID-19、俄罗斯对乌克兰的侵略、大屠杀、气候变化和 LGBTQ + 相关辩论这五个话题上，研究了聊天机器人对真实、虚假和模糊陈述的评估。我们使用英语、俄语和乌克兰语的提示，比较聊天机器人在高资源语言和低资源语言环境下的表现。此外，我们利用定义为导向的提示，探索了聊天机器人按照政治传播概念（如虚假信息、错误信息和阴谋论）对陈述进行评估的能力。我们还系统地测试了来源偏见对这种评估的影响，通过将具体声明归属于不同的政治和社会行为者来模拟来源偏见。结果显示，ChatGPT 在基准真实性评估任务中表现出较高的性能，在没有预训练的情况下，在不同语言之间平均评估准确性为 72%。Bing Chat 的准确率为 67%。我们观察到聊天机器人如何在高资源语言和低资源语言中评估提示，并且如何根据政治传播概念调整其评估，ChatGPT 提供的输出比 Bing Chat 更为细致入微。最后，我们发现聊天机器人在某些与真实性检测相关的任务中的性能因陈述的话题或归属的来源而有所变化。这些发现突显了基于 LLM 的聊天机器人在解决在线环境中不同形式的虚假信息方面的潜力，但也指出了由于特定因素（如提示语言或话题）导致其潜力实现方式的重大差异。

信赖生成 AI：聊天机器人能有效验证政治信息吗？

In Generative AI we Trust: Can Chatbots Effectively Verify Political  Information?

In this paper, we present the Logically submissions to De-Factify 2 challenge
(DE-FACTIFY 2023) on the task 1 of Multi-Modal Fact Checking. We describes our
submissions to this challenge including explored evidence retrieval and
selection techniques, pre-trained cross-modal and unimodal models, and a
cross-modal veracity model based on the well established Transformer Encoder
(TE) architecture which is heavily relies on the concept of self-attention.
Exploratory analysis is also conducted on this Factify 2 data set that uncovers
the salient multi-modal patterns and hypothesis motivating the architecture
proposed in this work. A series of preliminary experiments were done to
investigate and benchmarking different pre-trained embedding models, evidence
retrieval settings and thresholds. The final system, a standard two-stage
evidence based veracity detection system, yields weighted avg. 0.79 on both val
set and final blind test set on the task 1, which achieves 3rd place with a
small margin to the top performing system on the leaderboard among 9
participants.

本文描述了我们在多模态事实检查挑战中的提交，包括探索的证据检索和选择技术，预训练的跨模态和单模型，以及基于 Transformer Encoder 体系结构的跨模态准确性模型等内容，并在 Factify 2 数据集上进行的探索性分析揭示了突出的多模态模式和假设。最终系统，实现了在任务一的加权平均值为 0.79 的性能和排名第三的成绩。

基于证据检索技术和 Transformer 编码架构的多模式事实核查系统 Factify2

Logically at Factify 2: A Multi-Modal Fact Checking System Based on Evidence Retrieval techniques and Transformer Encoder Architecture

Fake news is a growing challenge for social networks and media. Detection of
fake news always has been a problem for many years, but after the evolution of
social networks and increasing speed of news dissemination in recent years has
been considered again. There are several approaches to solving this problem,
one of which is to detect fake news based on its text style using deep neural
networks. In recent years, one of the most used forms of deep neural networks
for natural language processing is transfer learning with transformers. BERT is
one of the most promising transformers who outperforms other models in many NLP
benchmarks. This article, we introduce MWPBert, which uses two parallel BERT
networks to perform veracity detection on full-text news articles. One of the
BERT networks encodes news headline, and another encodes news body. Since the
input length of the BERT network is limited and constant and the news body is
usually a long text, we cannot fed the whole news text into the BERT.
Therefore, using the MaxWorth algorithm, we selected the part of the news text
that is more valuable for fact-checking, and fed it into the BERT network.
Finally, we encode the output of the two BERT networks to an output network to
classify the news. The experiment results showed that the proposed model
outperformed previous models in terms of accuracy and other performance
measures.

介绍了一种使用两个并行 BERT 网络来对全文新闻文章进行真实性检测的新方法 MWPBert，通过 MaxWorth 算法选择更有价值的新闻文本部分，最终将两个 BERT 网络的输出编码为一个输出网络，实验结果表明该模型在准确性和其他性能指标方面优于以前的模型。