The real-time processing of time series signals is a critical issue for many
real-life applications. The idea of real-time processing is especially
important in audio domain as the human perception of sound is sensitive to any
kind of disturbance in perceived signals, especially the lag between auditory
and visual modalities. The rise of deep learning (DL) models complicated the
landscape of signal processing. Although they often have superior quality
compared to standard DSP methods, this advantage is diminished by higher
latency. In this work we propose novel method for minimization of inference
time latency and memory consumption, called Short-Term Memory Convolution
(STMC) and its transposed counterpart. The main advantage of STMC is the low
latency comparable to long short-term memory (LSTM) networks. Furthermore, the
training of STMC-based models is faster and more stable as the method is based
solely on convolutional neural networks (CNNs). In this study we demonstrate an
application of this solution to a U-Net model for a speech separation task and
GhostNet model in acoustic scene classification (ASC) task. In case of speech
separation we achieved a 5-fold reduction in inference time and a 2-fold
reduction in latency without affecting the output quality. The inference time
for ASC task was up to 4 times faster while preserving the original accuracy.

本文提出了一种名为 Short-Term Memory Convolution（STMC）的卷积神经网络方法，用于音频领域中的实时处理，该方法能够以低延迟比 LSTM 网络更稳定快速地进行训练和推理，实现了语音分离和声场分类的更快速度和更高的准确性。