With the ever-growing amounts of textual data from a large variety of
languages, domains, and genres, it has become standard to evaluate NLP
algorithms on multiple datasets in order to ensure consistent performance
across heterogeneous setups. However, such multiple comparisons pose
significant challenges to traditional statistical analysis methods in NLP and
can lead to erroneous conclusions. In this paper, we propose a Replicability
Analysis framework for a statistically sound analysis of multiple comparisons
between algorithms for NLP tasks. We discuss the theoretical advantages of this
framework over the current, statistically unjustified, practice in the NLP
literature, and demonstrate its empirical value across four applications:
multi-domain dependency parsing, multilingual POS tagging, cross-domain
sentiment classification and word similarity prediction.

该论文提出了一种重复性分析框架，用于对 NLP 任务算法的多个比较进行统计学分析，相对于 NLP 文献中目前未经统计证明的传统做法具有显著的理论优势，并通过多个应用案例展示了其实证价值。