Engagement analysis finds various applications in healthcare, education,
advertisement, services. Deep Neural Networks, used for analysis, possess
complex architecture and need large amounts of input data, computational power,
inference time. These constraints challenge embedding systems into devices for
real-time use. To address these limitations, we present a novel two-stream
feature fusion "Tensor-Convolution and Convolution-Transformer Network"
(TCCT-Net) architecture. To better learn the meaningful patterns in the
temporal-spatial domain, we design a "CT" stream that integrates a hybrid
convolutional-transformer. In parallel, to efficiently extract rich patterns
from the temporal-frequency domain and boost processing speed, we introduce a
"TC" stream that uses Continuous Wavelet Transform (CWT) to represent
information in a 2D tensor form. Evaluated on the EngageNet dataset, the
proposed method outperforms existing baselines, utilizing only two behavioral
features (head pose rotations) compared to the 98 used in baseline models.
Furthermore, comparative analysis shows TCCT-Net's architecture offers an
order-of-magnitude improvement in inference speed compared to state-of-the-art
image-based Recurrent Neural Network (RNN) methods. The code will be released
at this https URL

为了解决在设备实时使用时由于复杂架构、大量输入数据、计算能力和推理时间等限制，将嵌入系统的挑战，本研究提出了一种新颖的两流特征融合 “张量卷积和卷积变换网络” (TCCT-Net) 架构。该架构在 EngageNet 数据集上得到了验证，相比基准模型只使用两个行为特征（头姿势旋转），提出的方法表现出更好的性能。此外，对比分析显示 TCCT-Net 架构在推理速度方面相较于最先进的基于图像的循环神经网络 (RNN) 方法获得了数量级的提升。