Deep Learning Hard (DL-HARD) is a new annotated dataset designed to more
effectively evaluate neural ranking models on complex topics. It builds on TREC
Deep Learning (DL) topics by extensively annotating them with question intent
categories, answer types, wikified entities, topic categories, and result type
metadata from a commercial web search engine. Based on this data, we introduce
a framework for identifying challenging queries. DL-HARD contains fifty topics
from the official DL 2019/2020 evaluation benchmark, half of which are newly
and independently assessed. We perform experiments using the official submitted
runs to DL on DL-HARD and find substantial differences in metrics and the
ranking of participating systems. Overall, DL-HARD is a new resource that
promotes research on neural ranking methods by focusing on challenging and
complex topics.

介绍了一个名为 DL-HARD 的数据集，它通过对 TREC Deep Learning (DL) 主题进行广泛注释，从而更有效地评估神经排序模型，包括问题意图分类、答案类型、话题类别、来自商业网络搜索引擎的实体以及结果类型元数据等，同时还介绍了识别挑战性查询的框架。DL-HARD 包含了 DL 2019/2020 评估基准的 50 个主题，其中一半是新的并且独立评估的。该研究通过在 DL-HARD 上进行实验，发现参与系统的评估指标和排名存在实质性差异，从而推动了神经排序方法的研究。