Data annotation, the practice of assigning descriptive labels to raw data, is
pivotal in optimizing the performance of machine learning models. However, it
is a resource-intensive process susceptible to biases introduced by annotators.
The emergence of sophisticated Large Language Models (LLMs), like ChatGPT
presents a unique opportunity to modernize and streamline this complex
procedure. While existing research extensively evaluates the efficacy of LLMs,
as annotators, this paper delves into the biases present in LLMs, specifically
GPT 3.5 and GPT 4o when annotating hate speech data. Our research contributes
to understanding biases in four key categories: gender, race, religion, and
disability. Specifically targeting highly vulnerable groups within these
categories, we analyze annotator biases. Furthermore, we conduct a
comprehensive examination of potential factors contributing to these biases by
scrutinizing the annotated data. We introduce our custom hate speech detection
dataset, HateSpeechCorpus, to conduct this research. Additionally, we perform
the same experiments on the ETHOS (Mollas et al., 2022) dataset also for
comparative analysis. This paper serves as a crucial resource, guiding
researchers and practitioners in harnessing the potential of LLMs for
dataannotation, thereby fostering advancements in this critical field. The
HateSpeechCorpus dataset is available here:
this https URL

通过分析 GPT 3.5 和 GPT 4o 在标注仇恨言论数据时存在的偏见，本研究对性别、种族、宗教和残疾等四个主要类别中高度脆弱的群体进行了偏见分析，并通过对标注数据的仔细审查，全面研究了可能导致这些偏见的潜在因素。该研究为指导研究人员和从业者充分利用大语言模型进行数据标注，从而推进这一关键领域的进展提供重要资源。