State-of-the-art neural network language models (NNLMs) represented by long
short term memory recurrent neural networks (LSTM-RNNs) and Transformers are
becoming highly complex. They are prone to overfitting and poor generalization
when given limited training data. To this end, an overarching full Bayesian
learning framework encompassing three methods is proposed in this paper to
account for the underlying uncertainty in LSTM-RNN and Transformer LMs. The
uncertainty over their model parameters, choice of neural activations and
hidden output representations are modeled using Bayesian, Gaussian Process and
variational LSTM-RNN or Transformer LMs respectively. Efficient inference
approaches were used to automatically select the optimal network internal
components to be Bayesian learned using neural architecture search. A minimal
number of Monte Carlo parameter samples as low as one was also used. These
allow the computational costs incurred in Bayesian NNLM training and evaluation
to be minimized. Experiments are conducted on two tasks: AMI meeting
transcription and Oxford-BBC LipReading Sentences 2 (LRS2) overlapped speech
recognition using state-of-the-art LF-MMI trained factored TDNN systems
featuring data augmentation, speaker adaptation and audio-visual multi-channel
beamforming for overlapped speech. Consistent performance improvements over the
baseline LSTM-RNN and Transformer LMs with point estimated model parameters and
drop-out regularization were obtained across both tasks in terms of perplexity
and word error rate (WER). In particular, on the LRS2 data, statistically
significant WER reductions up to 1.3% and 1.2% absolute (12.1% and 11.3%
relative) were obtained over the baseline LSTM-RNN and Transformer LMs
respectively after model combination between Bayesian NNLMs and their
respective baselines.

提出一种普遍的贝叶斯学习框架，利用三种方法模拟 LSTM-RNN 和 Transformer LMs 的模型参数、神经激活选择和隐藏输出表示的不确定性，使用神经结构搜索自动选择优化网络内部组件，并使用有效的推断方法来降低计算成本，并在 AMI 会议转录和 LRS2 超叠话音频道的语音识别任务中实验验证，相对于基线 LSTM-RNN 和 Transformer LMs，在困惑度和字错误率方面均取得了一致的性能提升。