Pattern recognition based on RGB-Event data is a newly arising research topic
and previous works usually learn their features using CNN or Transformer. As we
know, CNN captures the local features well and the cascaded self-attention
mechanisms are good at extracting the long-range global relations. It is
intuitive to combine them for high-performance RGB-Event based video
recognition, however, existing works fail to achieve a good balance between the
accuracy and model parameters, as shown in Fig.~\ref{firstimage}. In this work,
we propose a novel RGB-Event based recognition framework termed TSCFormer,
which is a relatively lightweight CNN-Transformer model. Specifically, we
mainly adopt the CNN as the backbone network to first encode both RGB and Event
data. Meanwhile, we initialize global tokens as the input and fuse them with
RGB and Event features using the BridgeFormer module. It captures the global
long-range relations well between both modalities and maintains the simplicity
of the whole model architecture at the same time. The enhanced features will be
projected and fused into the RGB and Event CNN blocks, respectively, in an
interactive manner using F2E and F2V modules. Similar operations are conducted
for other CNN blocks to achieve adaptive fusion and local-global feature
enhancement under different resolutions. Finally, we concatenate these three
features and feed them into the classification head for pattern recognition.
Extensive experiments on two large-scale RGB-Event benchmark datasets
(PokerEvent and HARDVS) fully validated the effectiveness of our proposed
TSCFormer. The source code and pre-trained models will be released at
this https URL

基于 RGB-Event 数据的模式识别是一个新兴的研究课题，本文提出了一种相对轻量级的 CNN-Transformer 模型 TSCFormer，该模型在全局和本地特征提取上取得了平衡，经过大规模实验证实了其有效性。