What are the limits of automated Twitter sentiment classification? We analyze
a large set of manually labeled tweets in different languages, use them as
training data, and construct automated classification models. It turns out that
the quality of classification models depends much more on the quality and size
of training data than on the type of the model trained. Experimental results
indicate that there is no statistically significant difference between the
performance of the top classification models. We quantify the quality of
training data by applying various annotator agreement measures, and identify
the weakest points of different datasets. We show that the model performance
approaches the inter-annotator agreement when the size of the training set is
sufficiently large. However, it is crucial to regularly monitor the self- and
inter-annotator agreements since this improves the training datasets and
consequently the model performance. Finally, we show that there is strong
evidence that humans perceive the sentiment classes (negative, neutral, and
positive) as ordered.

该研究分析使用不同语言的大量手动标记的推文作为训练数据构建自动分类模型时，训练数据的质量和数量对于分类模型的质量比所使用的模型类型更为重要，并且当训练集的大小足够大时，模型的性能接近于标注者一致性，但定期监视标注者的一致性是非常重要的，从而改善训练数据集和模型性能，最后我们还表明，有强有力的证据表明，人们认为情感类别（消极的，中性的和积极的）是有序的。