Starting with early successes in computer vision tasks, deep learning based
techniques have since overtaken state of the art approaches in a multitude of
domains. However, it has been demonstrated time and again that these techniques
fail to capture semantic context and logical constraints, instead often relying
on spurious correlations to arrive at the answer. Since application of deep
learning techniques to critical scenarios are dependent on adherence to domain
specific constraints, several attempts have been made to address this issue.
One limitation holding back a thorough exploration of this area, is a lack of
suitable datasets which feature a rich set of rules. In order to address this,
we present the ChessVision Dataset, consisting of 200,000+ images of annotated
chess games in progress, requiring recreation of the game state from its
corresponding image. This is accompanied by a curated set of rules which
constrains the set of predictions to "reasonable" game states, and are designed
to probe key semantic abilities like localization and enumeration. Alongside
standard metrics, additional metrics to measure performance with regards to
logical consistency is presented. We analyze several popular and state of the
art vision models on this task, and show that, although their performance on
standard metrics are laudable, they produce a plethora of incoherent results,
indicating that this dataset presents a significant challenge for future works.

我们介绍了一个数据集，命名为 ChessVision 数据集，其中包含 200,000 + 张带有注释的象棋游戏进行中的图像，需要从图像中重新创建游戏状态，并附带一套约束以确保产生 “合理” 的游戏状态，同时还提出了用于衡量逻辑一致性的附加指标。通过对该任务进行分析，我们发现现有的视觉模型在标准指标上表现可观，但产生了大量不连贯的结果，这表明此数据集对未来研究提出了重大挑战。