Crowdsourced machine learning on competition platforms such as Kaggle is a
popular and often effective method for generating accurate models. Typically,
teams vie for the most accurate model, as measured by overall error on a
holdout set, and it is common towards the end of such competitions for teams at
the top of the leaderboard to ensemble or average their models outside the
platform mechanism to get the final, best global model. In arXiv:2201.10408,
the authors developed an alternative crowdsourcing framework in the context of
fair machine learning, in order to integrate community feedback into models
when subgroup unfairness is present and identifiable. There, unlike in
classical crowdsourced ML, participants deliberately specialize their efforts
by working on subproblems, such as demographic subgroups in the service of
fairness. Here, we take a broader perspective on this work: we note that within
this framework, participants may both specialize in the service of fairness and
simply to cater to their particular expertise (e.g., focusing on identifying
bird species in an image classification task). Unlike traditional
crowdsourcing, this allows for the diversification of participants' efforts and
may provide a participation mechanism to a larger range of individuals (e.g. a
machine learning novice who has insight into a specific fairness concern). We
present the first medium-scale experimental evaluation of this framework, with
46 participating teams attempting to generate models to predict income from
American Community Survey data. We provide an empirical analysis of teams'
approaches, and discuss the novel system architecture we developed. From here,
we give concrete guidance for how best to deploy such a framework.

通过竞赛平台的众包机器学习是一种流行且常见的方法，这篇论文开发了一种另类的众包框架来整合社区反馈，以解决群体不公平问题，并提供了关于部署这种框架的具体指导。

多样化集成：众包机器学习实验

Diversified Ensembling: An Experiment in Crowdsourced Machine Learning

User contributions in the form of posts, comments, and votes are essential to
the success of online communities. However, allowing user participation also
invites undesirable behavior such as trolling. In this paper, we characterize
antisocial behavior in three large online discussion communities by analyzing
users who were banned from these communities. We find that such users tend to
concentrate their efforts in a small number of threads, are more likely to post
irrelevantly, and are more successful at garnering responses from other users.
Studying the evolution of these users from the moment they join a community up
to when they get banned, we find that not only do they write worse than other
users over time, but they also become increasingly less tolerated by the
community. Further, we discover that antisocial behavior is exacerbated when
community feedback is overly harsh. Our analysis also reveals distinct groups
of users with different levels of antisocial behavior that can change over
time. We use these insights to identify antisocial users early on, a task of
high practical importance to community maintainers.

本研究通过对三个大型在线讨论社区被封禁用户的分析，表征了不良行为。我们发现这些用户更容易发表与主题无关的内容，并更容易引起其他用户的反应，而实施不良行为通常发生在社区反馈过度苛刻的情况下。此外，我们还发现这些用户的排斥度随着时间的推移而增加，更易被社区禁言。 研究结果表明，在线社区维护者会从这些信息中获得帮助，从而更早地发现和限制不受欢迎的行为。