Crowdsourced machine learning on competition platforms such as Kaggle is a
popular and often effective method for generating accurate models. Typically,
teams vie for the most accurate model, as measured by overall error on a
holdout set, and it is common towards the end of such competitions for teams at
the top of the leaderboard to ensemble or average their models outside the
platform mechanism to get the final, best global model. In arXiv:2201.10408,
the authors developed an alternative crowdsourcing framework in the context of
fair machine learning, in order to integrate community feedback into models
when subgroup unfairness is present and identifiable. There, unlike in
classical crowdsourced ML, participants deliberately specialize their efforts
by working on subproblems, such as demographic subgroups in the service of
fairness. Here, we take a broader perspective on this work: we note that within
this framework, participants may both specialize in the service of fairness and
simply to cater to their particular expertise (e.g., focusing on identifying
bird species in an image classification task). Unlike traditional
crowdsourcing, this allows for the diversification of participants' efforts and
may provide a participation mechanism to a larger range of individuals (e.g. a
machine learning novice who has insight into a specific fairness concern). We
present the first medium-scale experimental evaluation of this framework, with
46 participating teams attempting to generate models to predict income from
American Community Survey data. We provide an empirical analysis of teams'
approaches, and discuss the novel system architecture we developed. From here,
we give concrete guidance for how best to deploy such a framework.

通过竞赛平台的众包机器学习是一种流行且常见的方法，这篇论文开发了一种另类的众包框架来整合社区反馈，以解决群体不公平问题，并提供了关于部署这种框架的具体指导。