This work introduces a new multispectral database and novel approaches for
eyeblink detection in RGB and Near-Infrared (NIR) individual images. Our
contributed dataset (mEBAL2, multimodal Eye Blink and Attention Level
estimation, Version 2) is the largest existing eyeblink database, representing
a great opportunity to improve data-driven multispectral approaches for blink
detection and related applications (e.g., attention level estimation and
presentation attack detection in face biometrics). mEBAL2 includes 21,100 image
sequences from 180 different students (more than 2 million labeled images in
total) while conducting a number of e-learning tasks of varying difficulty or
taking a real course on HTML initiation through the edX MOOC platform. mEBAL2
uses multiple sensors, including two Near-Infrared (NIR) and one RGB camera to
capture facial gestures during the execution of the tasks, as well as an
Electroencephalogram (EEG) band to get the cognitive activity of the user and
blinking events. Furthermore, this work proposes a Convolutional Neural Network
architecture as benchmark for blink detection on mEBAL2 with performances up to
97%. Different training methodologies are implemented using the RGB spectrum,
NIR spectrum, and the combination of both to enhance the performance on
existing eyeblink detectors. We demonstrate that combining NIR and RGB images
during training improves the performance of RGB eyeblink detectors (i.e.,
detection based only on a RGB image). Finally, the generalization capacity of
the proposed eyeblink detectors is validated in wilder and more challenging
environments like the HUST-LEBW dataset to show the usefulness of mEBAL2 to
train a new generation of data-driven approaches for eyeblink detection.

这项研究介绍了一个新的多光谱数据库和针对 RGB 和近红外（NIR）个体图像的眨眼检测的新方法。通过提供的数据集（mEBAL2），我们有机会改进数据驱动的多光谱方法，以提高眨眼检测和相关应用的表现。通过在各种难度的在线学习任务中使用多个传感器捕捉面部动作的方式，以及结合脑电图（EEG）频段记录用户的认知活动和眨眼事件，我们证明了这些方法的泛化能力。最终，通过在更具挑战性的环境中验证模型的适用性，我们展示了训练 mEBAL2 可以用于培养新一代数据驱动的眨眼检测方法的重要性。

mEBAL2 数据库与基准：基于图像的多光谱眨眼检测

mEBAL2 Database and Benchmark: Image-based Multispectral Eyeblink  Detection

Effective and real-time eyeblink detection is of wide-range applications,
such as deception detection, drive fatigue detection, face anti-spoofing, etc.
Although numerous of efforts have already been paid, most of them focus on
addressing the eyeblink detection problem under the constrained indoor
conditions with the relative consistent subject and environment setup.
Nevertheless, towards the practical applications eyeblink detection in the wild
is more required, and of greater challenges. However, to our knowledge this has
not been well studied before. In this paper, we shed the light to this research
topic. A labelled eyeblink in the wild dataset (i.e., HUST-LEBW) of 673
eyeblink video samples (i.e., 381 positives, and 292 negatives) is first
established by us. These samples are captured from the unconstrained movies,
with the dramatic variation on human attribute, human pose, illumination
condition, imaging configuration, etc. Then, we formulate eyeblink detection
task as a spatial-temporal pattern recognition problem. After locating and
tracking human eye using SeetaFace engine and KCF tracker respectively, a
modified LSTM model able to capture the multi-scale temporal information is
proposed to execute eyeblink verification. A feature extraction approach that
reveals appearance and motion characteristics simultaneously is also proposed.
The experiments on HUST-LEBW reveal the superiority and efficiency of our
approach. It also verifies that, the existing eyeblink detection methods cannot
achieve satisfactory performance in the wild.

本文研究实时眨眼检测技术在野外环境中的应用。作者提出了一个人眼位置和轨迹跟踪后使用改进的 LSTM 模型识别多尺度时序信息的眨眼检测方法，并建立了适用于野外环境的眨眼数据集（HUST-LEBW）以验证该方法的可行性和效率。