When solving decision and optimisation problems, many competing algorithms
(model and solver choices) have complementary strengths. Typically, there is no
single algorithm that works well for all instances of a problem. Automated
algorithm selection has been shown to work very well for choosing a suitable
algorithm for a given instance. However, the cost of training can be
prohibitively large due to running candidate algorithms on a representative set
of training instances. In this work, we explore reducing this cost by choosing
a subset of the training instances on which to train. We approach this problem
in three ways: using active learning to decide based on prediction uncertainty,
augmenting the algorithm predictors with a timeout predictor, and collecting
training data using a progressively increasing timeout. We evaluate
combinations of these approaches on six datasets from ASLib and present the
reduction in labelling cost achieved by each option.

从 ASLib 中的六个数据集中评估了三种方法：基于预测不确定性的主动学习、使用超时预测器增强算法预测器以及使用逐步增加超时的方式收集训练数据，并展示了每个选项所实现的标记成本的降低。