Human annotator simulation (HAS) serves as a cost-effective substitute for human evaluation such as data annotation and system assessment. Human perception and behaviour during human evaluation exhibit inherent variability due to diverse cognitive processes and subjective interpretations, which should be taken into account in modelling to better mimic the way people perceive and interact with the world. This paper introduces a novel meta-learning framework that treats HAS as a zero-shot density estimation problem, which incorporates human variability and allows for the efficient generation of human-like annotations for unlabelled test inputs. Under this framework, we propose two new model classes, conditional integer flows and conditional softmax flows, to account for ordinal and categorical annotations, respectively. The proposed method is evaluated on three real-world human evaluation tasks and shows superior capability and efficiency to predict the aggregated behaviours of human annotators, match the distribution of human annotations, and simulate the inter-annotator disagreements.

本研究提出了一种新的元学习框架，将人类注释器模拟作为一种零样本密度估计问题，以更好地模拟人们感知和与世界互动的方式，该方法在三个真实世界的人类评估任务上展示了出色的能力和效率，以预测人类注释者的行为、与人类注释的分布相匹配以及模拟注释者之间的不一致性。

必须是主观的：通过零样本密度估计进行人类注释者模拟