By leveraging the power of Large Language Models(LLMs) and speech foundation
models, state of the art speech-text bimodal works can achieve challenging
tasks like spoken translation(ST) and question answering(SQA) altogether with
much simpler architectures. In this paper, we utilize the capability of Whisper
encoder and pre-trained Yi-6B. Empirical results reveal that modal alignment
can be achieved with one layer module and hundred hours of speech-text
multitask corpus. We further swap the Yi-6B with human preferences aligned
version of Yi-6B-Chat during inference, and discover that the alignment
capability is applicable as well. In addition, the alignment subspace revealed
by singular value decomposition(SVD) also implies linear alignment subspace is
sparse, which leaves the possibility to concatenate other features like
voice-print or video to expand modality.

利用大型语言模型（LLMs）和语音基础模型，先进的语音 - 文本双模工作可以实现复杂的任务，如口语翻译（ST）和问题回答（SQA），同时具有更简单的结构。本文利用 Whisper 编码器和预训练的 Yi-6B 的功能，通过经验结果发现，使用一个层的模块和百小时的语音 - 文本多任务语料库可以实现模态对齐。我们在推理过程中进一步交换 Yi-6B 为与人类偏好相符的 Yi-6B-Chat 版本，发现模态对齐能力同样适用。此外，奇异值分解（SVD）揭示的对齐子空间还意味着线性对齐子空间是稀疏的，这为连接其他特征（如声纹或视频）以扩展模态性留下了可能性。