Spiking neural networks (SNNs) have demonstrated the capability to achieve
comparable performance to deep neural networks (DNNs) in both visual and
linguistic domains while offering the advantages of improved energy efficiency
and adherence to biological plausibility. However, the extension of such
single-modality SNNs into the realm of multimodal scenarios remains an
unexplored territory. Drawing inspiration from the concept of contrastive
language-image pre-training (CLIP), we introduce a novel framework, named
SpikeCLIP, to address the gap between two modalities within the context of
spike-based computing through a two-step recipe involving ``Alignment
Pre-training + Dual-Loss Fine-tuning". Extensive experiments demonstrate that
SNNs achieve comparable results to their DNN counterparts while significantly
reducing energy consumption across a variety of datasets commonly used for
multimodal model evaluation. Furthermore, SpikeCLIP maintains robust
performance in image classification tasks that involve class labels not
predefined within specific categories.

通过双步骤配准预训练与双损失微调，该研究介绍了一种名为 SpikeCLIP 的新框架，用于解决基于脉冲计算的两种模态之间的差异，结果表明 SNN 在多模态模型评估中显著减少能耗，并且在包含未在特定类别中预定义的类标签的图像分类任务中保持稳健的性能。

SpikeCLIP：对比性语言 - 图像预训练脉冲神经网络

SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural  Network

Multimodal knowledge graphs (MKGs), which intuitively organize information in
various modalities, can benefit multiple practical downstream tasks, such as
recommendation systems, and visual question answering. However, most MKGs are
still far from complete, which motivates the flourishing of MKG reasoning
models. Recently, with the development of general artificial architectures, the
pretrained transformer models have drawn increasing attention, especially for
multimodal scenarios. However, the research of multimodal pretrained
transformer (MPT) for knowledge graph reasoning (KGR) is still at an early
stage. As the biggest difference between MKG and other multimodal data, the
rich structural information underlying the MKG still cannot be fully leveraged
in existing MPT models. Most of them only utilize the graph structure as a
retrieval map for matching images and texts connected with the same entity.
This manner hinders their reasoning performances. To this end, we propose the
graph Structure Guided Multimodal Pretrained Transformer for knowledge graph
reasoning, termed SGMPT. Specifically, the graph structure encoder is adopted
for structural feature encoding. Then, a structure-guided fusion module with
two different strategies, i.e., weighted summation and alignment constraint, is
first designed to inject the structural information into both the textual and
visual features. To the best of our knowledge, SGMPT is the first MPT model for
multimodal KGR, which mines the structural information underlying the knowledge
graph. Extensive experiments on FB15k-237-IMG and WN18-IMG, demonstrate that
our SGMPT outperforms existing state-of-the-art models, and prove the
effectiveness of the designed strategies.

我们提出了图结构引导的多模态预训练变换器（SGMPT）用于知识图谱推理，它采用图结构编码器来进行结构特征编码，并采用加权求和和对齐约束两种不同策略的结构引导融合模块，将结构信息注入文本和视觉特征，实验证明了 SGMPT 在多模态 KGR 方面的效果优于现有的最先进模型。

基于结构导向的多模态预训练变压器用于知识图谱推理

Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph  Reasoning

Emotional Reaction Intensity(ERI) estimation is an important task in
multimodal scenarios, and has fundamental applications in medicine, safe
driving and other fields. In this paper, we propose a solution to the ERI
challenge of the fifth Affective Behavior Analysis in-the-wild(ABAW), a
dual-branch based multi-output regression model. The spatial attention is used
to better extract visual features, and the Mel-Frequency Cepstral Coefficients
technology extracts acoustic features, and a method named modality dropout is
added to fusion multimodal features. Our method achieves excellent results on
the official validation set.

本文提出了一种解决第五届野外情感行为分析（ABAW）中 Emotional Reaction Intensity（ERI）挑战的双分支基于多输出回归模型，利用空间关注更好地提取视觉特征，利用 Mel-Frequency Cepstral Coefficients 技术提取声学特征，并添加了一种名为模态丢失的融合多模态特征的方法，在官方验证集上取得了优秀的结果。