We propose Audio-Visual Lightweight ITerative model (AVLIT), an effective and
lightweight neural network that uses Progressive Learning (PL) to perform
audio-visual speech separation in noisy environments. To this end, we adopt the
Asynchronous Fully Recurrent Convolutional Neural Network (A-FRCNN), which has
shown successful results in audio-only speech separation. Our architecture
consists of an audio branch and a video branch, with iterative A-FRCNN blocks
sharing weights for each modality. We evaluated our model in a controlled
environment using the NTCD-TIMIT dataset and in-the-wild using a synthetic
dataset that combines LRS3 and WHAM!. The experiments demonstrate the
superiority of our model in both settings with respect to various audio-only
and audio-visual baselines. Furthermore, the reduced footprint of our model
makes it suitable for low resource applications.

本文提出了一种名为 AVLIT 的轻量级神经网络，使用渐进学习（PL）在嘈杂的环境中进行音视频语音分离。通过异步完全循环卷积神经网络（A-FRCNN）的结构分支，我们的模型在音频和视觉两个分支下进行迭代学习，并在音频和视觉分离方面表现出较高的精度。同时由于模型尺寸小，因此适合低资源环境应用。