Contrasting Language-image pertaining (CLIP) has recently shown promising
open-world and few-shot performance on 2D image-based recognition tasks.
However, the transferred capability of CLIP to the novel event camera data
still remains under-explored. In particular, due to the modality gap with the
image-text data and the lack of large-scale datasets, achieving this goal is
non-trivial and thus requires significant research innovation. In this paper,
we propose E-CLIP, a novel and effective framework that unleashes the potential
of CLIP for event-based recognition to compensate for the lack of large-scale
event-based datasets. Our work addresses two crucial challenges: 1) how to
generalize CLIP's visual encoder to event data while fully leveraging events'
unique properties, e.g., sparsity and high temporal resolution; 2) how to
effectively align the multi-modal embeddings, i.e., image, text, and events. To
this end, we first introduce a novel event encoder that subtly models the
temporal information from events and meanwhile generates event prompts to
promote the modality bridging. We then design a text encoder that generates
content prompts and utilizes hybrid text prompts to enhance the E-CLIP's
generalization ability across diverse datasets. With the proposed event
encoder, text encoder, and original image encoder, a novel Hierarchical Triple
Contrastive Alignment (HTCA) module is introduced to jointly optimize the
correlation and enable efficient knowledge transfer among the three modalities.
We conduct extensive experiments on two recognition benchmarks, and the results
demonstrate that our E-CLIP outperforms existing methods by a large margin of
+3.94% and +4.62% on the N-Caltech dataset, respectively, in both fine-tuning
and few-shot settings. Moreover, our E-CLIP can be flexibly extended to the
event retrieval task using both text or image queries, showing plausible
performance.

提出了一种名为 E-CLIP 的框架，它通过引入新颖的事件编码器和文本编码器，以及原始图像编码器，采用层次三重对比对齐 (HTCA) 模块来优化相关性，实现三种模态之间的知识传递，并在 N-Caltech 数据集上在微调和少样本设置下将性能提升了 3.94% 和 4.62%。此外，E-CLIP 可以灵活地扩展到使用文本或图像查询的事件检索任务。

E-CLIP: 通过 CLIP 实现高效标签的事件驱动开放式理解

E-CLIP: Towards Label-efficient Event-based Open-world Understanding by  CLIP

Complex event retrieval is a challenging research problem, especially when no
training videos are available. An alternative to collecting training videos is
to train a large semantic concept bank a priori. Given a text description of an
event, event retrieval is performed by selecting concepts linguistically
related to the event description and fusing the concept responses on unseen
videos. However, defining an exhaustive concept lexicon and pre-training it
requires vast computational resources. Therefore, recent approaches automate
concept discovery and training by leveraging large amounts of weakly annotated
web data. Compact visually salient concepts are automatically obtained by the
use of concept pairs or, more generally, n-grams. However, not all visually
salient n-grams are necessarily useful for an event query--some combinations of
concepts may be visually compact but irrelevant--and this drastically affects
performance. We propose an event retrieval algorithm that constructs pairs of
automatically discovered concepts and then prunes those concepts that are
unlikely to be helpful for retrieval. Pruning depends both on the query and on
the specific video instance being evaluated. Our approach also addresses
calibration and domain adaptation issues that arise when applying concept
detectors to unseen videos. We demonstrate large improvements over other vision
based systems on the TRECVID MED 13 dataset.

通过构建配对概念，通过消除不可能对检索有帮助的概念来修剪，该方法解决了自动概念发现和培训中出现的问题，并通过解决视觉和领域适应问题展示了在 TRECVID MED 13 数据集上相对于其他基于视觉的系统的大幅度改进。