Treebank selection for parsing evaluation and the spurious effects that might
arise from a biased choice have not been explored in detail. This paper studies
how evaluating on a single subset of treebanks can lead to weak conclusions.
First, we take a few contrasting parsers, and run them on subsets of treebanks
proposed in previous work, whose use was justified (or not) on criteria such as
typology or data scarcity. Second, we run a large-scale version of this
experiment, create vast amounts of random subsets of treebanks, and compare on
them many parsers whose scores are available. The results show substantial
variability across subsets and that although establishing guidelines for good
treebank selection is hard, it is possible to detect potentially harmful
strategies.

研究了树库选择对解析评估的影响，发现使用单个树库子集进行评估可能会导致不准确的结论。实验发现，树库选择具有很大的变异性，虽然确定良好的树库选择指南很困难，但可以检测出潜在的有害策略。

多树库句法分析评估的脆弱性

The Fragility of Multi-Treebank Parsing Evaluation

We present a graph-based Tree Adjoining Grammar (TAG) parser that uses
BiLSTMs, highway connections, and character-level CNNs. Our best end-to-end
parser, which jointly performs supertagging, POS tagging, and parsing,
outperforms the previously reported best results by more than 2.2 LAS and UAS
points. The graph-based parsing architecture allows for global inference and
rich feature representations for TAG parsing, alleviating the fundamental
trade-off between transition-based and graph-based parsing systems. We also
demonstrate that the proposed parser achieves state-of-the-art performance in
the downstream tasks of Parsing Evaluation using Textual Entailments (PETE) and
Unbounded Dependency Recovery. This provides further support for the claim that
TAG is a viable formalism for problems that require rich structural analysis of
sentences.

本文介绍了基于图形的 Tree Adjoining Grammar (TAG) 解析器，并使用 BiLSTMs、Highway 连接和字符级 CNNs 来实现，该解析器在全局推理和特征表示方面具有先进性，在短语结构随从词性标注和解析方面性能显著优于以往的结果，并且在解析评估和无界依赖恢复方面实现了最先进的性能。