Event-based semantic segmentation (ESS) is a fundamental yet challenging task
for event camera sensing. The difficulties in interpreting and annotating event
data limit its scalability. While domain adaptation from images to event data
can help to mitigate this issue, there exist data representational differences
that require additional effort to resolve. In this work, for the first time, we
synergize information from image, text, and event-data domains and introduce
OpenESS to enable scalable ESS in an open-world, annotation-efficient manner.
We achieve this goal by transferring the semantically rich CLIP knowledge from
image-text pairs to event streams. To pursue better cross-modality adaptation,
we propose a frame-to-event contrastive distillation and a text-to-event
semantic consistency regularization. Experimental results on popular ESS
benchmarks showed our approach outperforms existing methods. Notably, we
achieve 53.93% and 43.31% mIoU on DDD17 and DSEC-Semantic without using either
event or frame labels.

本研究介绍了 OpenESS，利用图像、文本和事件数据领域的信息协同工作，以实现可扩展的事件感知的语义分割任务，通过跨模态适应和语义一致性正则化，得到了优于现有方法的实验结果。