We introduce HyperLex - a dataset and evaluation resource that quantifies the
extent of of the semantic category membership, that is, type-of relation also
known as hyponymy-hypernymy or lexical entailment (LE) relation between 2,616
concept pairs. Cognitive psychology research has established that typicality
and category/class membership are computed in human semantic memory as a
gradual rather than binary relation. Nevertheless, most NLP research, and
existing large-scale invetories of concept category membership (WordNet,
DBPedia, etc.) treat category membership and LE as binary. To address this, we
asked hundreds of native English speakers to indicate typicality and strength
of category membership between a diverse range of concept pairs on a
crowdsourcing platform. Our results confirm that category membership and LE are
indeed more gradual than binary. We then compare these human judgements with
the predictions of automatic systems, which reveals a huge gap between human
performance and state-of-the-art LE, distributional and representation learning
models, and substantial differences between the models themselves. We discuss a
pathway for improving semantic models to overcome this discrepancy, and
indicate future application areas for improved graded LE systems.

HyperLex 是一份数据集和评估资源，旨在量化语义类别成员资格的范围，即类型关系，也称为下位词 - 上位词或词汇蕴含关系。通过将数百名英语母语者作为研究对象，我们证实了类别成员资格和下位词 - 上位词关系更多是渐进的而非二元的。通过将人的研究成果和自动系统的预测作比较，我们发现两者之间存在巨大的差距，讨论了改进语义模型的途径，并指出了改进评分 LE 系统的未来应用领域。