With the rise of deep learning and intelligent vehicle, the smart assistant
has become an essential in-car component to facilitate driving and provide
extra functionalities. In-car smart assistants should be able to process
general as well as car-related commands and perform corresponding actions,
which eases driving and improves safety. However, there is a data scarcity
issue for low resource languages, hindering the development of research and
applications. In this paper, we introduce a new dataset, Cantonese In-car
Audio-Visual Speech Recognition (CI-AVSR), for in-car command recognition in
the Cantonese language with both video and audio data. It consists of 4,984
samples (8.3 hours) of 200 in-car commands recorded by 30 native Cantonese
speakers. Furthermore, we augment our dataset using common in-car background
noises to simulate real environments, producing a dataset 10 times larger than
the collected one. We provide detailed statistics of both the clean and the
augmented versions of our dataset. Moreover, we implement two multimodal
baselines to demonstrate the validity of CI-AVSR. Experiment results show that
leveraging the visual signal improves the overall performance of the model.
Although our best model can achieve a considerable quality on the clean test
set, the speech recognition quality on the noisy data is still inferior and
remains as an extremely challenging task for real in-car speech recognition
systems. The dataset and code will be released at
this https URL.

本文提出了一个新的中文粤语数据集 - Cantonese In-car Audio-Visual Speech Recognition (CI-AVSR)，用于研究车内指令识别。使用视觉和音频数据来识别汽车相关的指令，并采用常见的车内背景噪声进行数据增强来模拟真实环境，同时还实现了两种多模态基线以证明数据集的有效性和可行性。