By leveraging the power of Large Language Models(LLMs) and speech foundation
models, state of the art speech-text bimodal works can achieve challenging
tasks like spoken translation(ST) and question answering(SQA) altogether with
much simpler architectures. In this paper, we utilize the capability of Whisper
encoder and pre-trained Yi-6B. Empirical results reveal that modal alignment
can be achieved with one layer module and hundred hours of speech-text
multitask corpus. We further swap the Yi-6B with human preferences aligned
version of Yi-6B-Chat during inference, and discover that the alignment
capability is applicable as well. In addition, the alignment subspace revealed
by singular value decomposition(SVD) also implies linear alignment subspace is
sparse, which leaves the possibility to concatenate other features like
voice-print or video to expand modality.

利用大型语言模型（LLMs）和语音基础模型，先进的语音 - 文本双模工作可以实现复杂的任务，如口语翻译（ST）和问题回答（SQA），同时具有更简单的结构。本文利用 Whisper 编码器和预训练的 Yi-6B 的功能，通过经验结果发现，使用一个层的模块和百小时的语音 - 文本多任务语料库可以实现模态对齐。我们在推理过程中进一步交换 Yi-6B 为与人类偏好相符的 Yi-6B-Chat 版本，发现模态对齐能力同样适用。此外，奇异值分解（SVD）揭示的对齐子空间还意味着线性对齐子空间是稀疏的，这为连接其他特征（如声纹或视频）以扩展模态性留下了可能性。

跨可转移的语音转文大型语言模型对齐模块

Transferable speech-to-text large language model alignment module

Our contribution introduces a groundbreaking multimodal large language model
designed to comprehend multi-images, multi-audio, and multi-images-multi-audio
within a single multiturn session. Leveraging state-of-the-art models, we
utilize the SigLIP encoder for visual inputs and the Whisper Encoder for audio
inputs. Notably, this multimodal large language model is bilingual, proficient
in understanding both English and Malay simultaneously. We proudly unveil two
versions of this model: TinyLlama with 1.1B parameters, and Mistral with 7B
parameters. With its ability to navigate diverse modalities and languages, our
model represents a significant advancement for the Malaysian context and
beyond.
All models released at
this https URL

我们介绍了一种开创性的多模态大型语言模型，能够在一个多轮对话中理解多图像、多音频和多图像 - 多音频。借助最先进的模型，我们利用 SigLIP 编码器进行视觉输入和 Whisper 编码器进行音频输入。值得注意的是，这个多模态大型语言模型是双语的，能够同时理解英文和马来文。我们自豪地推出了这个模型的两个版本：参数量为 1.1B 的 TinyLlama 和参数量为 7B 的 Mistral。我们的模型能够处理多样的模态和语言，代表了马来西亚及其他地区的重大进展。

MMMModal -- 多图像 多音频 多轮 多模态

MMMModal -- Multi-Images Multi-Audio Multi-turn Multi-Modal

Pre-trained speech encoders have been central to pushing state-of-the-art
results across various speech understanding and generation tasks. Nonetheless,
the capabilities of these encoders in low-resource settings are yet to be
thoroughly explored. To address this, we conduct a comprehensive set of
experiments using a representative set of 3 state-of-the-art encoders
(Wav2vec2, WavLM, Whisper) in the low-resource setting across 7 speech
understanding and generation tasks. We provide various quantitative and
qualitative analyses on task performance, convergence speed, and
representational properties of the encoders. We observe a connection between
the pre-training protocols of these encoders and the way in which they capture
information in their internal layers. In particular, we observe the Whisper
encoder exhibits the greatest low-resource capabilities on content-driven tasks
in terms of performance and convergence speed.

使用三种最先进的语音编码器，Wav2vec2、WavLM 和 Whisper，对于 7 个语音理解和生成任务在低资源环境下进行了全面的实验，研究结果表明，Whisper 编码器在执行与内容相关的任务时具有最好的性能和收敛速度。