The integration of different modalities, such as audio and visual
information, plays a crucial role in human perception of the surrounding
environment. Recent research has made significant progress in designing fusion
modules for audio-visual speech separation. However, they predominantly focus
on multi-modal fusion architectures situated either at the top or bottom
positions, rather than comprehensively considering multi-modal fusion at
various hierarchical positions within the network. In this paper, we propose a
novel model called self- and cross-attention network (SCANet), which leverages
the attention mechanism for efficient audio-visual feature fusion. SCANet
consists of two types of attention blocks: self-attention (SA) and
cross-attention (CA) blocks, where the CA blocks are distributed at the top
(TCA), middle (MCA) and bottom (BCA) of SCANet. These blocks maintain the
ability to learn modality-specific features and enable the extraction of
different semantics from audio-visual features. Comprehensive experiments on
three standard audio-visual separation benchmarks (LRS2, LRS3, and VoxCeleb2)
demonstrate the effectiveness of SCANet, outperforming existing
state-of-the-art (SOTA) methods while maintaining comparable inference time.

提出了一种名为自注意力网络（SCANet）的模型，通过利用注意机制实现有效的音频 - 视觉特征融合，其中包含了自注意力块（SA）和交叉注意力块（CA），能够从音频 - 视觉特征中提取不同的语义信息，并在三个标准音频 - 视觉分离基准上得到了更好的效果。