Engagement measurement finds application in healthcare, education,
advertisement, and services. The use of physiological and behavioral features
is viable, but the impracticality of traditional physiological measurement
arises due to the need for contact sensors. We demonstrate the feasibility of
unsupervised remote photoplethysmography (rPPG) as an alternative for contact
sensors in deriving heart rate variability (HRV) features, then fusing these
with behavioral features to measure engagement in online group meetings.
Firstly, a unique Engagement Dataset of online interactions among social
workers is collected with granular engagement labels, offering insight into
virtual meeting dynamics. Secondly, a pre-trained rPPG model is customized to
reconstruct accurate rPPG signals from video meetings in an unsupervised
manner, enabling the calculation of HRV features. Thirdly, the feasibility of
estimating engagement from HRV features using short observation windows, with a
notable enhancement when using longer observation windows of two to four
minutes, is demonstrated. Fourthly, the effectiveness of behavioral cues is
evaluated and fused with physiological data, which further enhances engagement
estimation performance. An accuracy of 94% is achieved when only HRV features
are used, eliminating the need for contact sensors or ground truth signals. The
incorporation of behavioral cues raises the accuracy to 96%. Facial video
analysis offers precise engagement measurement, beneficial for future
applications.

使用无监督的远程光电容积脉搏图 (rPPG) 技术，结合行为特征，有效地测量在线团体会议中的参与度，通过使用仅心率变异性 (HRV) 特征可实现 94% 的准确性，结合行为线索后可提高至 96% 的准确性。

使用行为特征的无监督远程光电容积测量分析在线会议参与者的参与度

Analyzing Participants' Engagement during Online Meetings Using  Unsupervised Remote Photoplethysmography with Behavioral Features

Engagement in virtual learning is crucial for a variety of factors including
learner satisfaction, performance, and compliance with learning programs, but
measuring it is a challenging task. There is therefore considerable interest in
utilizing artificial intelligence and affective computing to measure engagement
in natural settings as well as on a large scale. This paper introduces a novel,
privacy-preserving method for engagement measurement from videos. It uses
facial landmarks, which carry no personally identifiable information, extracted
from videos via the MediaPipe deep learning solution. The extracted facial
landmarks are fed to a Spatial-Temporal Graph Convolutional Network (ST-GCN) to
output the engagement level of the learner in the video. To integrate the
ordinal nature of the engagement variable into the training process, ST-GCNs
undergo training in a novel ordinal learning framework based on transfer
learning. Experimental results on two video student engagement measurement
datasets show the superiority of the proposed method compared to previous
methods with improved state-of-the-art on the EngageNet dataset with a %3.1
improvement in four-class engagement level classification accuracy and on the
Online Student Engagement dataset with a %1.5 improvement in binary engagement
classification accuracy. The relatively lightweight ST-GCN and its integration
with the real-time MediaPipe deep learning solution make the proposed approach
capable of being deployed on virtual learning platforms and measuring
engagement in real time.

介绍一种通过人工智能和情感计算来测量虚拟学习中学生参与度的隐私保护方法，该方法利用从视频中提取的没有个人身份信息的面部标志点，通过 Spatial-Temporal Graph Convolutional Network (ST-GCN) 模型输出学生在视频中的参与度水平。