Contrasting Language-image pertaining (CLIP) has recently shown promising
open-world and few-shot performance on 2D image-based recognition tasks.
However, the transferred capability of CLIP to the novel event camera data
still remains under-explored. In particular, due to the modality gap with the
image-text data and the lack of large-scale datasets, achieving this goal is
non-trivial and thus requires significant research innovation. In this paper,
we propose E-CLIP, a novel and effective framework that unleashes the potential
of CLIP for event-based recognition to compensate for the lack of large-scale
event-based datasets. Our work addresses two crucial challenges: 1) how to
generalize CLIP's visual encoder to event data while fully leveraging events'
unique properties, e.g., sparsity and high temporal resolution; 2) how to
effectively align the multi-modal embeddings, i.e., image, text, and events. To
this end, we first introduce a novel event encoder that subtly models the
temporal information from events and meanwhile generates event prompts to
promote the modality bridging. We then design a text encoder that generates
content prompts and utilizes hybrid text prompts to enhance the E-CLIP's
generalization ability across diverse datasets. With the proposed event
encoder, text encoder, and original image encoder, a novel Hierarchical Triple
Contrastive Alignment (HTCA) module is introduced to jointly optimize the
correlation and enable efficient knowledge transfer among the three modalities.
We conduct extensive experiments on two recognition benchmarks, and the results
demonstrate that our E-CLIP outperforms existing methods by a large margin of
+3.94% and +4.62% on the N-Caltech dataset, respectively, in both fine-tuning
and few-shot settings. Moreover, our E-CLIP can be flexibly extended to the
event retrieval task using both text or image queries, showing plausible
performance.

提出了一种名为 E-CLIP 的框架，它通过引入新颖的事件编码器和文本编码器，以及原始图像编码器，采用层次三重对比对齐 (HTCA) 模块来优化相关性，实现三种模态之间的知识传递，并在 N-Caltech 数据集上在微调和少样本设置下将性能提升了 3.94% 和 4.62%。此外，E-CLIP 可以灵活地扩展到使用文本或图像查询的事件检索任务。