Peer review is a core element of the scientific process, particularly in conference-centered fields such as ML and NLP. However, only few studies have evaluated its properties empirically. Aiming to fill this gap, we present a corpus that contains over 4k reviews and 1.2k author responses from ACL-2018. We quantitatively and qualitatively assess the corpus. This includes a pilot study on paper weaknesses given by reviewers and on quality of author responses. We then focus on the role of the rebuttal phase, and propose a novel task to predict after-rebuttal (i.e., final) scores from initial reviews and author responses. Although author responses do have a marginal (and statistically significant) influence on the final scores, especially for borderline papers, our results suggest that a reviewer's final score is largely determined by her initial score and the distance to the other reviewers' initial scores. In this context, we discuss the conformity bias inherent to peer reviewing, a bias that has largely been overlooked in previous research. We hope our analyses will help better assess the usefulness of the rebuttal phase in NLP conferences.

本研究通过对ACL-2018会议的4000余个论文的评论和1200余个作者回复的语料库的定量和定性分析以及针对构成论文弱点给出的试点研究和关于作者回复质量的定量和定性评估，重点放在了反驳阶段的角色以及从初审回复和作者回复中预测最终评分的新任务上，研究结果发现虽然作者回复会对最终评分产生一定的影响，但评审人的最终评分往往取决于他们的初审评分和与其他评审人初审评分之间的距离，而这种从众心理一直是同行评议中存在的偏见，但这个偏见在以前的研究中被忽视了。

我的反驳有意义吗？从一次重要的NLP会议中得到的启示