Event-based semantic segmentation (ESS) is a fundamental yet challenging task
for event camera sensing. The difficulties in interpreting and annotating event
data limit its scalability. While domain adaptation from images to event data
can help to mitigate this issue, there exist data representational differences
that require additional effort to resolve. In this work, for the first time, we
synergize information from image, text, and event-data domains and introduce
OpenESS to enable scalable ESS in an open-world, annotation-efficient manner.
We achieve this goal by transferring the semantically rich CLIP knowledge from
image-text pairs to event streams. To pursue better cross-modality adaptation,
we propose a frame-to-event contrastive distillation and a text-to-event
semantic consistency regularization. Experimental results on popular ESS
benchmarks showed our approach outperforms existing methods. Notably, we
achieve 53.93% and 43.31% mIoU on DDD17 and DSEC-Semantic without using either
event or frame labels.

本研究介绍了 OpenESS，利用图像、文本和事件数据领域的信息协同工作，以实现可扩展的事件感知的语义分割任务，通过跨模态适应和语义一致性正则化，得到了优于现有方法的实验结果。

OpenESS: 基于事件的开放语义场景理解

OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies

Event-based semantic segmentation has gained popularity due to its capability
to deal with scenarios under high-speed motion and extreme lighting conditions,
which cannot be addressed by conventional RGB cameras. Since it is hard to
annotate event data, previous approaches rely on event-to-image reconstruction
to obtain pseudo labels for training. However, this will inevitably introduce
noise, and learning from noisy pseudo labels, especially when generated from a
single source, may reinforce the errors. This drawback is also called
confirmation bias in pseudo-labeling. In this paper, we propose a novel hybrid
pseudo-labeling framework for unsupervised event-based semantic segmentation,
HPL-ESS, to alleviate the influence of noisy pseudo labels. In particular, we
first employ a plain unsupervised domain adaptation framework as our baseline,
which can generate a set of pseudo labels through self-training. Then, we
incorporate offline event-to-image reconstruction into the framework, and
obtain another set of pseudo labels by predicting segmentation maps on the
reconstructed images. A noisy label learning strategy is designed to mix the
two sets of pseudo labels and enhance the quality. Moreover, we propose a soft
prototypical alignment module to further improve the consistency of target
domain features. Extensive experiments show that our proposed method
outperforms existing state-of-the-art methods by a large margin on the
DSEC-Semantic dataset (+5.88% accuracy, +10.32% mIoU), which even surpasses
several supervised methods.

提出了一种新型的混合伪标签框架（HPL-ESS），用于减轻噪声伪标签的影响，并通过软原型对齐模块进一步改善目标领域特征的一致性，实验证明该方法在 DSEC-Semantic 数据集上的准确性和平均交并比（mIoU）等性能优于现有的最先进方法。