We propose a new active learning (AL) framework, Active Learning++, which can utilize an annotator's labels as well as its rationale. Annotators can provide their rationale for choosing a label by ranking input features based on their importance for a given query. To incorporate this additional input, we modified the disagreement measure for a bagging-based Query by Committee (QBC) sampling strategy. Instead of weighing all committee models equally to select the next instance, we assign higher weight to the committee model with higher agreement with the annotator's ranking. Specifically, we generated a feature importance-based local explanation for each committee model. The similarity score between feature rankings provided by the annotator and the local model explanation is used to assign a weight to each corresponding committee model. This approach is applicable to any kind of ML model using model-agnostic techniques to generate local explanation such as LIME. With a simulation study, we show that our framework significantly outperforms a QBC based vanilla AL framework.

研究提出一种新的主动学习框架，称为Active Learning++，其可以利用注释者的标签以及其理由，通过修改基于Bagging的Query by Committee (QBC)采样策略的不一致度量方法，将委员会模型的权重分配给具有更高一致性的注释者排序的委员会模型，在模拟研究中表明该框架显著优于QBC based vanilla AL framework。

Active Learning++：利用本地模型解释包含标注者原因