We conducted a human subject study of named entity recognition on a noisy
corpus of conversational music recommendation queries, with many irregular and
novel named entities. We evaluated the human NER linguistic behaviour in these
challenging conditions and compared it with the most common NER systems
nowadays, fine-tuned transformers. Our goal was to learn about the task to
guide the design of better evaluation methods and NER algorithms. The results
showed that NER in our context was quite hard for both human and algorithms
under a strict evaluation schema; humans had higher precision, while the model
higher recall because of entity exposure especially during pre-training; and
entity types had different error patterns (e.g. frequent typing errors for
artists). The released corpus goes beyond predefined frames of interaction and
can support future work in conversational music recommendation.

这项人类主题研究的目标是学习如何在有挑战性的情况下，引导设计更好的评估方法和 NER 算法，比较 NER 在有许多不规则和新颖的命名实体的对话音乐推荐查询语料库上的人类和算法表现，并分析了不同类型实体的错误模式。