In this paper we present the first steps towards hardening the science of
measuring AI systems, by adopting metrology, the science of measurement and its
application, and applying it to human (crowd) powered evaluations. We begin
with the intuitive observation that evaluating the performance of an AI system
is a form of measurement. In all other science and engineering disciplines, the
devices used to measure are called instruments, and all measurements are
recorded with respect to the characteristics of the instruments used. One does
not report mass, speed, or length, for example, of a studied object without
disclosing the precision (measurement variance) and resolution (smallest
detectable change) of the instrument used. It is extremely common in the AI
literature to compare the performance of two systems by using a crowd-sourced
dataset as an instrument, but failing to report if the performance difference
lies within the capability of that instrument to measure. To illustrate the
adoption of metrology to benchmark datasets we use the word similarity
benchmark WS353 and several previously published experiments that use it for
evaluation.

本文旨在将计量科学应用于人力（群体）评估中，以加强测量人工智能系统性能的科学，并以词汇相似性基准 WS353 和先前发布的使用它进行评估的实验为例，采用计量学来对基准数据集进行基准测试。