Event-based semantic segmentation (ESS) is a fundamental yet challenging task
for event camera sensing. The difficulties in interpreting and annotating event
data limit its scalability. While domain adaptation from images to event data
can help to mitigate this issue, there exist data representational differences
that require additional effort to resolve. In this work, for the first time, we
synergize information from image, text, and event-data domains and introduce
OpenESS to enable scalable ESS in an open-world, annotation-efficient manner.
We achieve this goal by transferring the semantically rich CLIP knowledge from
image-text pairs to event streams. To pursue better cross-modality adaptation,
we propose a frame-to-event contrastive distillation and a text-to-event
semantic consistency regularization. Experimental results on popular ESS
benchmarks showed our approach outperforms existing methods. Notably, we
achieve 53.93% and 43.31% mIoU on DDD17 and DSEC-Semantic without using either
event or frame labels.

本研究介绍了 OpenESS，利用图像、文本和事件数据领域的信息协同工作，以实现可扩展的事件感知的语义分割任务，通过跨模态适应和语义一致性正则化，得到了优于现有方法的实验结果。

OpenESS: 基于事件的开放语义场景理解

OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies

Vision-Language Models (VLMs), integrating diverse information from multiple
modalities, have shown remarkable success across various tasks. However,
deploying VLMs, comprising large-scale vision and language models poses
challenges in resource-constrained scenarios. While pruning followed by
finetuning offers a potential solution to maintain performance with smaller
model sizes, its application to VLMs remains relatively unexplored, presenting
two main questions: how to distribute sparsity across different
modality-specific models, and how to repair the performance of pruned sparse
VLMs. To answer the first question, we conducted preliminary studies on VLM
pruning and found that pruning vision models and language models with the same
sparsity ratios contribute to nearly optimal performance. For the second
question, unlike finetuning unimodal sparse models, sparse VLMs involve
cross-modality interactions, requiring specialized techniques for post-pruning
performance repair. Moreover, while parameter-efficient LoRA finetuning has
been proposed to repair the performance of sparse models, a significant
challenge of weights merging arises due to the incompatibility of dense LoRA
modules with sparse models that destroy the sparsity of pruned models. To
tackle these challenges, we propose to Repair Sparse Vision-Language Models via
Sparse Cross-modality Adaptation (RESSA). RESSA utilizes cross-modality
finetuning to enhance task-specific performance and facilitate knowledge
distillation from original dense models. Additionally, we introduce SparseLoRA,
which applies sparsity directly to LoRA weights, enabling seamless integration
with sparse models. Our experimental results validate the effectiveness of
RESSA, showcasing significant enhancements, such as an 11.3\% improvement under
2:4 sparsity and a remarkable 47.6\% enhancement under unstructured 70\%
sparsity.

通过罕见模型剪枝和稀疏模型修复解决了资源受限场景下部署大规模 Vision-Language Models 的挑战，提出了一种稀疏跨模态适应方法 (RESSA)，其中包括稀疏 LoRA 技术和交互模态微调，实现了显著的性能增强。

RESSA：通过稀疏跨模态适应修复稀疏视觉语言模型

RESSA: Repair Sparse Vision-Language Models via Sparse Cross-Modality  Adaptation

Due to the large cross-modality discrepancy between 2D sketches and 3D
shapes, retrieving 3D shapes by sketches is a significantly challenging task.
To address this problem, we propose a novel framework to learn a discriminative
deep cross-modality adaptation model in this paper. Specifically, we first
separately adopt two metric networks, following two deep convolutional neural
networks (CNNs), to learn modality-specific discriminative features based on an
importance-aware metric learning method. Subsequently, we explicitly introduce
a cross-modality transformation network to compensate for the divergence
between two modalities, which can transfer features of 2D sketches to the
feature space of 3D shapes. We develop an adversarial learning based method to
train the transformation model, by simultaneously enhancing the holistic
correlations between data distributions of two modalities, and mitigating the
local semantic divergences through minimizing a cross-modality mean discrepancy
term. Experimental results on the SHREC 2013 and SHREC 2014 datasets clearly
show the superior retrieval performance of our proposed model, compared to the
state-of-the-art approaches.

本文提出了一种基于深度卷积神经网络的新颖的跨模态适应模型形成方法，其中采用度量学习和对抗学习方法来对 2D 手绘图和 3D 物体进行信息的迁移和转化，能够有效提高物体检索的结果。