When using ultrasound video as input, Deep Neural Network-based Silent Speech
Interfaces usually rely on the whole image to estimate the spectral parameters
required for the speech synthesis step. Although this approach is quite
straightforward, and it permits the synthesis of understandable speech, it has
several disadvantages as well. Besides the inability to capture the relations
between close regions (i.e. pixels) of the image, this pixel-by-pixel
representation of the image is also quite uneconomical. It is easy to see that
a significant part of the image is irrelevant for the spectral parameter
estimation task as the information stored by the neighbouring pixels is
redundant, and the neural network is quite large due to the large number of
input features. To resolve these issues, in this study we train an autoencoder
neural network on the ultrasound image; the estimation of the spectral speech
parameters is done by a second DNN, using the activations of the bottleneck
layer of the autoencoder network as features. In our experiments, the proposed
method proved to be more efficient than the standard approach: the measured
normalized mean squared error scores were lower, while the correlation values
were higher in each case. Based on the result of a listening test, the
synthesized utterances also sounded more natural to native speakers. A further
advantage of our proposed approach is that, due to the (relatively) small size
of the bottleneck layer, we can utilize several consecutive ultrasound images
during estimation without a significant increase in the network size, while
significantly increasing the accuracy of parameter estimation.

通过对超声图像训练自编码神经网络，将神经网络的输入特征从像素级别转移为自编码器的瓶颈层激活，以更高的效率和准确性估计语音合成所需的频谱参数。