Counter narratives - informed responses to hate speech contexts designed to
refute hateful claims and de-escalate encounters - have emerged as an effective
hate speech intervention strategy. While previous work has proposed automatic
counter narrative generation methods to aid manual interventions, the
evaluation of these approaches remains underdeveloped. Previous automatic
metrics for counter narrative evaluation lack alignment with human judgment as
they rely on superficial reference comparisons instead of incorporating key
aspects of counter narrative quality as evaluation criteria. To address prior
evaluation limitations, we propose a novel evaluation framework prompting LLMs
to provide scores and feedback for generated counter narrative candidates using
5 defined aspects derived from guidelines from counter narrative specialized
NGOs. We found that LLM evaluators achieve strong alignment to human-annotated
scores and feedback and outperform alternative metrics, indicating their
potential as multi-aspect, reference-free and interpretable evaluators for
counter narrative evaluation.

通过新的评估框架，我们提出了一种新颖的方法，让 LLMs 为生成的对抗叙事候选者提供评分和反馈，从而弥补以往评估方法的局限性，并显示出作为多方面、无参考和可解释的对抗叙事评估者的潜力。