In this paper, we propose three methods for generating synthetic samples to
train and evaluate multimodal large language models capable of processing both
text and speech inputs. Addressing the scarcity of samples containing both
modalities, synthetic data generation emerges as a crucial strategy to enhance
the performance of such systems and facilitate the modeling of cross-modal
relationships between the speech and text domains. Our process employs large
language models to generate textual components and text-to-speech systems to
generate speech components. The proposed methods offer a practical and
effective means to expand the training dataset for these models. Experimental
results show progress in achieving an integrated understanding of text and
speech. We also highlight the potential of using unlabeled speech data to
generate synthetic samples comparable in quality to those with available
transcriptions, enabling the expansion of these models to more languages.

我们提出了三种方法来生成合成样本，以训练和评估能够处理文本和语音输入的多模态大语言模型。通过解决包含多种模态的样本的稀缺性问题，合成数据生成成为提高这些系统性能并促进语音和文本领域的跨模态关系建模的关键策略。我们使用大型语言模型生成文本组件和文本到语音系统生成语音组件的过程。所提出的方法提供了一种实用且有效的扩展这些模型训练数据集的方式。实验结果表明，在理解文本和语音方面取得了进展。我们还强调了使用未标注的语音数据来生成质量可与有可用转录的样本媲美的合成样本的潜力，从而使这些模型能够更多地应用于其他语言。