This paper presents an augmentation of MSCOCO dataset where speech is added
to image and text. Speech captions are generated using text-to-speech (TTS)
synthesis resulting in 616,767 spoken captions (more than 600h) paired with
images. Disfluencies and speed perturbation are added to the signal in order to
sound more natural. Each speech signal (WAV) is paired with a JSON file
containing exact timecode for each word/syllable/phoneme in the spoken caption.
Such a corpus could be used for Language and Vision (LaVi) tasks including
speech input or output instead of text. Investigating multimodal learning
schemes for unsupervised speech pattern discovery is also possible with this
corpus, as demonstrated by a preliminary study conducted on a subset of the
corpus (10h, 10k spoken captions). The dataset is available on Zenodo:
this https URL

本研究介绍了 MSCOCO 数据集的增强版本，其中添加了语音和文本。使用文本转语音（TTS）合成生成语音说明，在语音信号中添加了不流畅和速度扰动以使其更加自然。此语料库可用于语言和视觉（LaVi）任务，其中包括语音输入或输出。同时在此数据集的一个子集上进行了初步的研究，探索了无监督语音模式发现的多模态学习方案。