Annually, research teams spend large amounts of money to evaluate the quality
of machine translation systems (WMT, inter alia). This is expensive because it
requires detailed human labor. The recently proposed annotation protocol, Error
Span Annotation (ESA), has annotators marking erroneous parts of the
translation. In our work, we help the annotators by pre-filling the span
annotations with automatic quality estimation. With AI assistance, we obtain
more detailed annotations while cutting down the time per span annotation by
half (71s/error span $\rightarrow$ 31s/error span). The biggest advantage of
ESA$^\mathrm{AI}$ protocol is an accurate priming of annotators (pre-filled
error spans) before they assign the final score as opposed to starting from
scratch. In addition, the annotation budget can be reduced by up to 24% with
filtering of examples that the AI deems to be very likely to be correct.

机器翻译系统的质量评估被提出的错误范围标注协议辅助，使用自动质量估计填充范围注释可以提供更详细的注释并将每个范围注释的时间缩短了一半，此外还可以减少最多 24% 的注释预算。

人工智能辅助机器翻译的人工评估

AI-Assisted Human Evaluation of Machine Translation

Online hate speech is a recent problem in our society that is rising at a
steady pace by leveraging the vulnerabilities of the corresponding regimes that
characterise most social media platforms. This phenomenon is primarily fostered
by offensive comments, either during user interaction or in the form of a
posted multimedia context. Nowadays, giant corporations own platforms where
millions of users log in every day, and protection from exposure to similar
phenomena appears to be necessary in order to comply with the corresponding
legislation and maintain a high level of service quality. A robust and reliable
system for detecting and preventing the uploading of relevant content will have
a significant impact on our digitally interconnected society. Several aspects
of our daily lives are undeniably linked to our social profiles, making us
vulnerable to abusive behaviours. As a result, the lack of accurate hate speech
detection mechanisms would severely degrade the overall user experience,
although its erroneous operation would pose many ethical concerns. In this
paper, we present 'ETHOS', a textual dataset with two variants: binary and
multi-label, based on YouTube and Reddit comments validated using the
Figure-Eight crowdsourcing platform. Furthermore, we present the annotation
protocol used to create this dataset: an active sampling procedure for
balancing our data in relation to the various aspects defined. Our key
assumption is that, even gaining a small amount of labelled data from such a
time-consuming process, we can guarantee hate speech occurrences in the
examined material.

在本文中，我们提出了基于 YouTube 和 Reddit 评论的文本数据集 'EHTOS'，并介绍了用于创建该数据集的注释协议，旨在平衡数据。我们的关键假设是，即使从这样耗时的过程中获得一小部分标记数据，我们也可以保证在所检查的材料中发现仇恨言论。

ETHOS：一个在线仇恨言论检测数据集

ETHOS: an Online Hate Speech Detection Dataset

This paper describes the AVA-Kinetics localized human actions video dataset.
The dataset is collected by annotating videos from the Kinetics-700 dataset
using the AVA annotation protocol, and extending the original AVA dataset with
these new AVA annotated Kinetics clips. The dataset contains over 230k clips
annotated with the 80 AVA action classes for each of the humans in key-frames.
We describe the annotation process and provide statistics about the new
dataset. We also include a baseline evaluation using the Video Action
Transformer Network on the AVA-Kinetics dataset, demonstrating improved
performance for action classification on the AVA test set. The dataset can be
downloaded from this https URL

该论文介绍了 AVA-Kinetics 本地化人类行为视频数据集，其中包含超过 230k 个帧，对每个人的关键帧进行了 80 个 AVA 动作类的注释，通过视频行动变换网络在 AVA-Kinetics 数据集上进行了基线评估，并证明了在 AVA 测试集上行动分类的表现有所改善。

AVA-Kinetics 本地化人类行为视频数据集

The AVA-Kinetics Localized Human Actions Video Dataset

Empathy, as defined in behavioral sciences, expresses the ability of human
beings to recognize, understand and react to emotions, attitudes and beliefs of
others. The lack of an operational definition of empathy makes it difficult to
measure it. In this paper, we address two related problems in automatic
affective behavior analysis: the design of the annotation protocol and the
automatic recognition of empathy from spoken conversations. We propose and
evaluate an annotation scheme for empathy inspired by the modal model of
emotions. The annotation scheme was evaluated on a corpus of real-life, dyadic
spoken conversations. In the context of behavioral analysis, we designed an
automatic segmentation and classification system for empathy. Given the
different speech and language levels of representation where empathy may be
communicated, we investigated features derived from the lexical and acoustic
spaces. The feature development process was designed to support both the fusion
and automatic selection of relevant features from high dimensional space. The
automatic classification system was evaluated on call center conversations
where it showed significantly better performance than the baseline.

本文提出并评估了一种情感识别标注方案，通过构建自动分割和分类系统，通过调查语言和声学空间中的特征来识别通话中的共情。该系统在呼叫中心对话中表现出明显的优异性能。