Human-in-the-loop optimization utilizes human expertise to guide machine
optimizers iteratively and search for an optimal solution in a solution space.
While prior empirical studies mainly investigated novices, we analyzed the
impact of the levels of expertise on the outcome quality and corresponding
subjective satisfaction. We conducted a study (N=60) in text, photo, and 3D
mesh optimization contexts. We found that novices can achieve an expert level
of quality performance, but participants with higher expertise led to more
optimization iteration with more explicit preference while keeping satisfaction
low. In contrast, novices were more easily satisfied and terminated faster.
Therefore, we identified that experts seek more diverse outcomes while the
machine reaches optimal results, and the observed behavior can be used as a
performance indicator for human-in-the-loop system designers to improve
underlying models. We inform future research to be cautious about the impact of
user expertise when designing human-in-the-loop systems.

该研究分析了人工智能优化过程中用户专业水平对解决方案质量和主观满意度的影响，从文本、照片和三维网格优化等不同领域进行了研究，认为专业人士更倾向于追求多样化的结果，这一行为可以用作设计者改进智能优化模型的绩效指标。

专家参与对于机器智能探究的影响

The Impact of Expertise in the Loop for Exploring Machine Rationality

Automated evaluation of open domain natural language generation (NLG) models
remains a challenge and widely used metrics such as BLEU and Perplexity can be
misleading in some cases. In our paper, we propose to evaluate natural language
generation models by learning to compare a pair of generated sentences by
fine-tuning BERT, which has been shown to have good natural language
understanding ability. We also propose to evaluate the model-level quality of
NLG models with sample-level comparison results with skill rating system. While
able to be trained in a fully self-supervised fashion, our model can be further
fine-tuned with a little amount of human preference annotation to better
imitate human judgment. In addition to evaluating trained models, we propose to
apply our model as a performance indicator during training for better
hyperparameter tuning and early-stopping. We evaluate our approach on both
story generation and chit-chat dialogue response generation. Experimental
results show that our model correlates better with human preference compared
with previous automated evaluation approaches. Training with the proposed
metric yields better performance in human evaluation, which further
demonstrates the effectiveness of the proposed model.

本文提出了一种使用 BERT fine-tuning 的方法来比较生成的自然语言模型，同时还提出了使用技能等级系统来评估模型质量，并将其作为性能指标，在训练过程中进行优化。实验结果表明，该方法与人类偏好的相关性更高，训练成果更优秀，有效性得到了证明。