Crowdsourcing information constitutes an important aspect of
human-in-the-loop learning for researchers across multiple disciplines such as
AI, HCI, and social science. While using crowdsourced data for subjective tasks
is not new, eliciting useful insights from such data remains challenging due to
a variety of factors such as difficulty of the task, personal prejudices of the
human evaluators, lack of question clarity, etc. In this paper, we consider one
such subjective evaluation task, namely that of estimating experienced emotions
of distressed individuals who are conversing with a human listener in an online
coaching platform. We explore strategies to aggregate the evaluators choices,
and show that a simple voting consensus is as effective as an optimum
aggregation method for the task considered. Intrigued by how an objective
assessment would compare to the subjective evaluation of evaluators, we also
designed a machine learning algorithm to perform the same task. Interestingly,
we observed a machine learning algorithm that is not explicitly modeled to
characterize evaluators' subjectivity is as reliable as the human evaluation in
terms of assessing the most dominant experienced emotions.

本文研究了在线辅导平台上对话的困扰个体的情绪估计的主观评估任务，探讨了聚合评估者选择的策略，展示了一个简单的投票共识与优化聚合方法在这个任务中同样有效，并设计了一种机器学习算法来执行相同的任务。有趣的是，我们观察到了一个没有明确建模评估者主观性的机器学习算法，在评估最主要的情绪时与人类评估一样可靠。