Decision support systems for classification tasks are predominantly designed
to predict the value of the ground truth labels. However, since their
predictions are not perfect, these systems also need to make human experts
understand when and how to use these predictions to update their own
predictions. Unfortunately, this has been proven challenging. In this context,
it has been recently argued that an alternative type of decision support
systems may circumvent this challenge. Rather than providing a single label
prediction, these systems provide a set of label prediction values constructed
using a conformal predictor, namely a prediction set, and forcefully ask
experts to predict a label value from the prediction set. However, the design
and evaluation of these systems have so far relied on stylized expert models,
questioning their promise. In this paper, we revisit the design of this type of
systems from the perspective of online learning and develop a methodology that
does not require, nor assumes, an expert model. Our methodology leverages the
nested structure of the prediction sets provided by any conformal predictor and
a natural counterfactual monotonicity assumption on the experts' predictions
over the prediction sets to achieve an exponential improvement in regret in
comparison with vanilla bandit algorithms. We conduct a large-scale human
subject study ($n = 2{,}751$) to verify our counterfactual monotonicity
assumption and compare our methodology to several competitive baselines. The
results suggest that decision support systems that limit experts' level of
agency may be practical and may offer greater performance than those allowing
experts to always exercise their own agency.

本文提出了一种新型决策支持系统，通过嵌套结构的预测集和专家的设想单调性假设，实现比普通 bandit 算法更好的结果；大规模人类学科研究 ($n=2751$) 表明，这种限制专家代理水平的决策支持系统实用且可能提供比允许专家始终行使其代理权的系统更好的性能。

使用反事实预测集设计决策支持系统

Designing Decision Support Systems Using Counterfactual Prediction Sets

We conducted a human subject study of named entity recognition on a noisy
corpus of conversational music recommendation queries, with many irregular and
novel named entities. We evaluated the human NER linguistic behaviour in these
challenging conditions and compared it with the most common NER systems
nowadays, fine-tuned transformers. Our goal was to learn about the task to
guide the design of better evaluation methods and NER algorithms. The results
showed that NER in our context was quite hard for both human and algorithms
under a strict evaluation schema; humans had higher precision, while the model
higher recall because of entity exposure especially during pre-training; and
entity types had different error patterns (e.g. frequent typing errors for
artists). The released corpus goes beyond predefined frames of interaction and
can support future work in conversational music recommendation.

这项人类主题研究的目标是学习如何在有挑战性的情况下，引导设计更好的评估方法和 NER 算法，比较 NER 在有许多不规则和新颖的命名实体的对话音乐推荐查询语料库上的人类和算法表现，并分析了不同类型实体的错误模式。

面向音乐推荐对话的命名实体识别研究

A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queries

To best assist human designers with different styles, Machine Learning (ML)
systems need to be able to adapt to them. However, there has been relatively
little prior work on how and when to best adapt an ML system to a co-designer.
In this paper we present threshold designer adaptation: a novel method for
adapting a creative ML model to an individual designer. We evaluate our
approach with a human subject study using a co-creative rhythm game design
tool. We find that designers prefer our proposed method and produce higher
quality content in comparison to an existing baseline.

本研究提出了阈值设计师适配方法，通过人体实验评估发现，设计师更喜欢该方法，相比于现有的基准控制组，他们创造的内容更具高质量。