Automated classifiers (ACs), often built via supervised machine learning
(SML), can categorize large, statistically powerful samples of data ranging
from text to images and video, and have become widely popular measurement
devices in communication science and related fields. Despite this popularity,
even highly accurate classifiers make errors that cause misclassification bias
and misleading results in downstream analyses-unless such analyses account for
these errors. As we show in a systematic literature review of SML applications,
communication scholars largely ignore misclassification bias. In principle,
existing statistical methods can use "gold standard" validation data, such as
that created by human annotators, to correct misclassification bias and produce
consistent estimates. We introduce and test such methods, including a new
method we design and implement in the R package misclassificationmodels, via
Monte Carlo simulations designed to reveal each method's limitations, which we
also release. Based on our results, we recommend our new error correction
method as it is versatile and efficient. In sum, automated classifiers, even
those below common accuracy standards or making systematic misclassifications,
can be useful for measurement with careful study design and appropriate error
correction methods.

通过系统文献综述，我们发现大多数通信学者没有考虑分类器的误分类偏差。我们介绍并测试了 “gold standard” 验证数据的错误校正方法，并通过蒙特卡罗模拟来揭示每种方法的局限性。我们推荐我们设计和实施的新方法作为更有效的误差校正方法。

自动内容分析中的错误分类导致回归偏差。我们能修复吗？是的我们可以！

Misclassification in Automated Content Analysis Causes Bias in  Regression. Can We Fix It? Yes We Can!

Hateful rhetoric is plaguing online discourse, fostering extreme societal
movements and possibly giving rise to real-world violence. A potential solution
to this growing global problem is citizen-generated counter speech where
citizens actively engage in hate-filled conversations to attempt to restore
civil non-polarized discourse. However, its actual effectiveness in curbing the
spread of hatred is unknown and hard to quantify. One major obstacle to
researching this question is a lack of large labeled data sets for training
automated classifiers to identify counter speech. Here we made use of a unique
situation in Germany where self-labeling groups engaged in organized online
hate and counter speech. We used an ensemble learning algorithm which pairs a
variety of paragraph embeddings with regularized logistic regression functions
to classify both hate and counter speech in a corpus of millions of relevant
tweets from these two groups. Our pipeline achieved macro F1 scores on out of
sample balanced test sets ranging from 0.76 to 0.97---accuracy in line and even
exceeding the state of the art. On thousands of tweets, we used crowdsourcing
to verify that the judgments made by the classifier are in close alignment with
human judgment. We then used the classifier to discover hate and counter speech
in more than 135,000 fully-resolved Twitter conversations occurring from 2013
to 2018 and study their frequency and interaction. Altogether, our results
highlight the potential of automated methods to evaluate the impact of
coordinated counter speech in stabilizing conversations on social media.

通过使用自我标记的群体进行有组织在线仇恨言论和反言论，研究使用集成学习算法识别有组织在线仇恨言论和反言论，发现自动化方法在评估社交媒体上协调反言论对稳定对话的影响的潜力。