This work builds together two popular blocks of neural architecture, namely
convolutional layers and Transformers, for large language models (LLMs).
Non-causal conformers are used ubiquitously in automatic speech recognition.
This work aims to adapt these architectures in a causal setup for training
LLMs. Transformers decoders effectively capture long-range dependencies over
several modalities and form a core backbone of modern advancements in machine
learning. Convolutional architectures have been popular in extracting features
in domains such as raw 1-D signals, speech, and images, to name a few. In this
paper, by combining local and global dependencies over latent representations
using causal convolutional filters and Transformer, we achieve significant
gains in performance. This work showcases a robust speech architecture that can
be integrated and adapted in a causal setup beyond speech applications for
large-scale language modeling.

本文将卷积层和 Transformer 这两个流行的神经网络块与大型语言模型（LLMs）进行整合。通过使用因果卷积滤波器和 Transformer 将潜在表示的局部和全局依赖相结合，取得了显著的性能提升。这项工作展示了一种强大的语音架构，可以在语言建模的大规模应用中进行整合和适应。