Although media bias detection is a complex multi-task problem, there is, to date, no unified benchmark grouping these evaluation tasks. We introduce the Media Bias Identification Benchmark (MBIB), a comprehensive benchmark that groups different types of media bias (e.g., linguistic, cognitive, political) under a common framework to test how prospective detection techniques generalize. After reviewing 115 datasets, we select nine tasks and carefully propose 22 associated datasets for evaluating media bias detection techniques. We evaluate MBIB using state-of-the-art Transformer techniques (e.g., T5, BART). Our results suggest that while hate speech, racial bias, and gender bias are easier to detect, models struggle to handle certain bias types, e.g., cognitive and political bias. However, our results show that no single technique can outperform all the others significantly. We also find an uneven distribution of research interest and resource allocation to the individual tasks in media bias. A unified benchmark encourages the development of more robust systems and shifts the current paradigm in media bias detection evaluation towards solutions that tackle not one but multiple media bias types simultaneously.

本研究介绍了 Media Bias Identification Benchmark（MBIB），这是一个包含不同类型媒体偏见的综合性基准，使用最新的 Transformer 技术对其进行了评估，结果表明，虽然仇恨言论、种族偏见和性别偏见更容易检测，但模型却无法处理某些偏见类型，同时该基准还发现了研究兴趣和资源分配的不均衡现象，鼓励开发更强大的系统来解决多个媒体偏见类型问题。

MBIB - 首个媒体偏见识别基准任务和数据集合集