We introduce the first open-domain dataset, called QuaRTz, for reasoning
about textual qualitative relationships. QuaRTz contains general qualitative
statements, e.g., "A sunscreen with a higher SPF protects the skin longer.",
twinned with 3864 crowdsourced situated questions, e.g., "Billy is wearing
sunscreen with a lower SPF than Lucy. Who will be best protected from the
sun?", plus annotations of the properties being compared. Unlike previous
datasets, the general knowledge is textual and not tied to a fixed set of
relationships, and tests a system's ability to comprehend and apply textual
qualitative knowledge in a novel setting. We find state-of-the-art results are
substantially (20%) below human performance, presenting an open challenge to
the NLP community.

介绍了第一个开放领域的定性关系数据集 QuaRTz，用于推理文本定性关系。QuaRTz 包含一般的定性陈述，结合 3864 个众包定位问题，对应的是类似 “Billy 涂有比 Lucy 更低 SPF 的防晒霜。谁的皮肤最好受保护？” 的问题，以及被比较对象的注释。与以往的数据集不同，一般知识是文本而不是固定的关系集，并且在新颖环境中测试系统理解和应用文本定性知识的能力。实验结果表明，最先进的结果比人类表现低得多（20％），这是自然语言处理社区的一个开放挑战。