The proliferation of misinformation and harmful narratives in online
discourse has underscored the critical need for effective Counter Narrative
(CN) generation techniques. However, existing automatic evaluation methods
often lack interpretability and fail to capture the nuanced relationship
between generated CNs and human perception. Aiming to achieve a higher
correlation with human judgments, this paper proposes a novel approach to asses
generated CNs that consists on the use of a Large Language Model (LLM) as a
evaluator. By comparing generated CNs pairwise in a tournament-style format, we
establish a model ranking pipeline that achieves a correlation of $0.88$ with
human preference. As an additional contribution, we leverage LLMs as zero-shot
(ZS) CN generators and conduct a comparative analysis of chat, instruct, and
base models, exploring their respective strengths and limitations. Through
meticulous evaluation, including fine-tuning experiments, we elucidate the
differences in performance and responsiveness to domain-specific data. We
conclude that chat-aligned models in ZS are the best option for carrying out
the task, provided they do not refuse to generate an answer due to security
concerns.

使用大型语言模型作为评估者，提出了一种对生成的反话进行评估的新方法，通过在锦标赛格式中对生成的反话进行成对比较，建立了一个模型排名流程，并与人类偏好达到了 0.88 的相关性。此外，通过细致的评估和性能分析，发现以聊天为导向的零 - shot 生成模型在此任务中是最佳选择，前提是它们不因安全问题而拒绝生成答案。