Benchmark datasets play an important role in evaluating Natural Language
Understanding (NLU) models. However, shortcuts -- unwanted biases in the
benchmark datasets -- can damage the effectiveness of benchmark datasets in
revealing models' real capabilities. Since shortcuts vary in coverage,
productivity, and semantic meaning, it is challenging for NLU experts to
systematically understand and avoid them when creating benchmark datasets. In
this paper, we develop a visual analytics system, ShortcutLens, to help NLU
experts explore shortcuts in NLU benchmark datasets. The system allows users to
conduct multi-level exploration of shortcuts. Specifically, Statistics View
helps users grasp the statistics such as coverage and productivity of shortcuts
in the benchmark dataset. Template View employs hierarchical and interpretable
templates to summarize different types of shortcuts. Instance View allows users
to check the corresponding instances covered by the shortcuts. We conduct case
studies and expert interviews to evaluate the effectiveness and usability of
the system. The results demonstrate that ShortcutLens supports users in gaining
a better understanding of benchmark dataset issues through shortcuts, inspiring
them to create challenging and pertinent benchmark datasets.

本文提出了一种名为 ShortcutLens 的可视化分析系统，帮助 NLU 专家探索 NLU 基准数据集中的捷径，该系统允许用户进行多层次的捷径探索，包括统计视图、模板视图和实例视图，研究结果表明 ShortcutLens 支持用户更好地了解基准数据集中的捷径问题，激励他们创建具有挑战性和相关性的基准数据集。