Recently, researchers have uncovered that neural retrieval models prefer
AI-generated content (AIGC), called source bias. Compared to active search
behavior, recommendation represents another important means of information
acquisition, where users are more prone to source bias. Furthermore, delving
into the recommendation scenario, as AIGC becomes integrated within the
feedback loop involving users, data, and the recommender system, it
progressively contaminates the candidate items, the user interaction history,
and ultimately, the data used to train the recommendation models. How and to
what extent the source bias affects the neural recommendation models within
feedback loop remains unknown. In this study, we extend the investigation of
source bias into the realm of recommender systems, specifically examining its
impact across different phases of the feedback loop. We conceptualize the
progression of AIGC integration into the recommendation content ecosystem in
three distinct phases-HGC dominate, HGC-AIGC coexist, and AIGC dominance-each
representing past, present, and future states, respectively. Through extensive
experiments across three datasets from diverse domains, we demonstrate the
prevalence of source bias and reveal a potential digital echo chamber with
source bias amplification throughout the feedback loop. This trend risks
creating a recommender ecosystem with limited information source, such as AIGC,
being disproportionately recommended. To counteract this bias and prevent its
escalation in the feedback loop, we introduce a black-box debiasing method that
maintains model impartiality towards both HGC and AIGC. Our experimental
results validate the effectiveness of the proposed debiasing method, confirming
its potential to disrupt the feedback loop.

研究表明在反馈循环中，源偏好对神经推荐模型产生了影响，因此引入了一种黑盒去偏方法以防止偏见在反馈循环中的升级。

信息回音室：探究在用户、数据和推荐系统反馈循环中的源偏见逐步升级

Source Echo Chamber: Exploring the Escalation of Source Bias in User,  Data, and Recommender System Feedback Loop

With the advancement of generation models, AI-generated content (AIGC) is
becoming more realistic, flooding the Internet. A recent study suggests that
this phenomenon has elevated the issue of source bias in text retrieval for web
searches. Specifically, neural retrieval models tend to rank generated texts
higher than human-written texts. In this paper, we extend the study of this
bias to cross-modal retrieval. Firstly, we successfully construct a suitable
benchmark to explore the existence of the bias. Subsequent extensive
experiments on this benchmark reveal that AI-generated images introduce an
invisible relevance bias to text-image retrieval models. Specifically, our
experiments show that text-image retrieval models tend to rank the AI-generated
images higher than the real images, even though the AI-generated images do not
exhibit more visually relevant features to the query than real images. This
invisible relevance bias is prevalent across retrieval models with varying
training data and architectures. Furthermore, our subsequent exploration
reveals that the inclusion of AI-generated images in the training data of the
retrieval models exacerbates the invisible relevance bias. The above phenomenon
triggers a vicious cycle, which makes the invisible relevance bias become more
and more serious. To elucidate the potential causes of invisible relevance and
address the aforementioned issues, we introduce an effective training method
aimed at alleviating the invisible relevance bias. Subsequently, we apply our
proposed debiasing method to retroactively identify the causes of invisible
relevance, revealing that the AI-generated images induce the image encoder to
embed additional information into their representation. This information
exhibits a certain consistency across generated images with different semantics
and can make the retriever estimate a higher relevance score.

在研究中，作者通过构建适当的基准来探索 AI 生成图像引起的不可见相关性偏差，这种偏差会使得文本 - 图像检索模型倾向于将 AI 生成的图像排在真实图像之前，然后提出了一种有效的训练方法来缓解这种不可见相关性偏差。

人工智能生成的图像为文本 - 图像检索引入了不可见的相关偏差

AI-Generated Images Introduce Invisible Relevance Bias to Text-Image  Retrieval

Though feature-alignment based Domain Adaptive Object Detection (DAOD) have
achieved remarkable progress, they ignore the source bias issue, i.e. the
aligned features are more favorable towards the source domain, leading to a
sub-optimal adaptation. Furthermore, the presence of domain shift between the
source and target domains exacerbates the problem of inconsistent
classification and localization in general detection pipelines. To overcome
these challenges, we propose a novel Distillation-based Unbiased Alignment
(DUA) framework for DAOD, which can distill the source features towards a more
balanced position via a pre-trained teacher model during the training process,
alleviating the problem of source bias effectively. In addition, we design a
Target-Relevant Object Localization Network (TROLN), which can mine
target-related knowledge to produce two classification-free metrics (IoU and
centerness). Accordingly, we implement a Domain-aware Consistency Enhancing
(DCE) strategy that utilizes these two metrics to further refine classification
confidences, achieving a harmonization between classification and localization
in cross-domain scenarios. Extensive experiments have been conducted to
manifest the effectiveness of this method, which consistently improves the
strong baseline by large margins, outperforming existing alignment-based works.

提出了基于蒸馏的无偏对齐 (Distillation-based Unbiased Alignment, DUA) 框架及目标相关对象定位网络 (Target-Relevant Object Localization Network, TROLN)，解决了领域自适应目标检测中源偏差和领域转移问题，通过蒸馏方式调整源特征位置，利用目标相关知识生成无需分类的两个度量指标 (IoU 和 centerness)，并采用领域感知的一致性增强策略 (DCE) 进一步改善分类一致性和定位准确性，在跨领域情景中显著提升性能，超过已有基于对齐的方法。

DUA-DA：基于蒸馏的无偏领域自适应目标检测

DUA-DA: Distillation-based Unbiased Alignment for Domain Adaptive Object  Detection

Recently, the emergence of large language models (LLMs) has revolutionized
the paradigm of information retrieval (IR) applications, especially in web
search. With their remarkable capabilities in generating human-like texts, LLMs
have created enormous texts on the Internet. As a result, IR systems in the
LLMs era are facing a new challenge: the indexed documents now are not only
written by human beings but also automatically generated by the LLMs. How these
LLM-generated documents influence the IR systems is a pressing and still
unexplored question. In this work, we conduct a quantitative evaluation of
different IR models in scenarios where both human-written and LLM-generated
texts are involved. Surprisingly, our findings indicate that neural retrieval
models tend to rank LLM-generated documents higher.We refer to this category of
biases in neural retrieval models towards the LLM-generated text as the
\textbf{source bias}. Moreover, we discover that this bias is not confined to
the first-stage neural retrievers, but extends to the second-stage neural
re-rankers. Then, we provide an in-depth analysis from the perspective of text
compression and observe that neural models can better understand the semantic
information of LLM-generated text, which is further substantiated by our
theoretical analysis.We also discuss the potential server concerns stemming
from the observed source bias and hope our findings can serve as a critical
wake-up call to the IR community and beyond. To facilitate future explorations
of IR in the LLM era, the constructed two new benchmarks and codes will later
be available at https://github.com/KID-22/LLM4IR-Bias.

大型语言模型在信息检索应用中产生的源偏见以及相关的模型评估和研究的重要性。