The volumetric representation of human interactions is one of the fundamental
domains in the development of immersive media productions and telecommunication
applications. Particularly in the context of the rapid advancement of Extended
Reality (XR) applications, this volumetric data has proven to be an essential
technology for future XR elaboration. In this work, we present a new multimodal
database to help advance the development of immersive technologies. Our
proposed database provides ethically compliant and diverse volumetric data, in
particular 27 participants displaying posed facial expressions and subtle body
movements while speaking, plus 11 participants wearing head-mounted displays
(HMDs). The recording system consists of a volumetric capture (VoCap) studio,
including 31 synchronized modules with 62 RGB cameras and 31 depth cameras. In
addition to textured meshes, point clouds, and multi-view RGB-D data, we use
one Lytro Illum camera for providing light field (LF) data simultaneously.
Finally, we also provide an evaluation of our dataset employment with regard to
the tasks of facial expression classification, HMDs removal, and point cloud
reconstruction. The dataset can be helpful in the evaluation and performance
testing of various XR algorithms, including but not limited to facial
expression recognition and reconstruction, facial reenactment, and volumetric
video. HEADSET and its all associated raw data and license agreement will be
publicly available for research purposes.

我们提出了一个新的多模态数据库来帮助推进沉浸技术的发展，该数据库提供了符合道德规范并且多样化的体积数据，可以支持面部表情识别等扩展现实算法的评估和性能测试。

头戴式耳机：部分遮挡的多模感知人类情感数据集

Headset: Human emotion awareness under partial occlusions multimodal  dataset

This work presents mEBAL, a multimodal database for eye blink detection and
attention level estimation. The eye blink frequency is related to the cognitive
activity and automatic detectors of eye blinks have been proposed for many
tasks including attention level estimation, analysis of neuro-degenerative
diseases, deception recognition, drive fatigue detection, or face
anti-spoofing. However, most existing databases and algorithms in this area are
limited to experiments involving only a few hundred samples and individual
sensors like face cameras. The proposed mEBAL improves previous databases in
terms of acquisition sensors and samples. In particular, three different
sensors are simultaneously considered: Near Infrared (NIR) and RGB cameras to
capture the face gestures and an Electroencephalography (EEG) band to capture
the cognitive activity of the user and blinking events. Regarding the size of
mEBAL, it comprises 6,000 samples and the corresponding attention level from 38
different students while conducting a number of e-learning tasks of varying
difficulty. In addition to presenting mEBAL, we also include preliminary
experiments on: i) eye blink detection using Convolutional Neural Networks
(CNN) with the facial images, and ii) attention level estimation of the
students based on their eye blink frequency.

这篇论文介绍了一个名为 mEBAL 的多模态数据库，用于眨眼检测和注意力水平估计，其改进了现有数据库在采集传感器和样本方面的限制，并基于学生眨眼频率估计其注意力水平，初步实验表明使用卷积神经网络在面部图像中眨眼检测的效果比较好。

mEBAL: 用于眨眼检测和注意力估计的多模态数据库

mEBAL: A Multimodal Database for Eye Blink Detection and Attention Level  Estimation

Efficient similarity retrieval from large-scale multimodal database is
pervasive in modern search engines and social networks. To support queries
across content modalities, the system should enable cross-modal correlation and
computation-efficient indexing. While hashing methods have shown great
potential in achieving this goal, current attempts generally fail to learn
isomorphic hash codes in a seamless scheme, that is, they embed multiple
modalities in a continuous isomorphic space and separately threshold embeddings
into binary codes, which incurs substantial loss of retrieval accuracy. In this
paper, we approach seamless multimodal hashing by proposing a novel Composite
Correlation Quantization (CCQ) model. Specifically, CCQ jointly finds
correlation-maximal mappings that transform different modalities into
isomorphic latent space, and learns composite quantizers that convert the
isomorphic latent features into compact binary codes. An optimization framework
is devised to preserve both intra-modal similarity and inter-modal correlation
through minimizing both reconstruction and quantization errors, which can be
trained from both paired and partially paired data in linear time. A
comprehensive set of experiments clearly show the superior effectiveness and
efficiency of CCQ against the state of the art hashing methods for both
unimodal and cross-modal retrieval.

本文提出了一种新的复合相关量化（CCQ）模型，通过最大化相关性的映射，将不同模态嵌入到连续的同构空间中，并同时学习复合量化器，将同构潜在特征转换为紧凑的二进制码，以实现高效的相似性检索。实验证明，CCQ 模型优于现有的哈希方法，可用于单模态和跨模态检索。