Recent end-to-end approaches have shown promise in extending large language
models (LLMs) to speech inputs, but face limitations in directly assessing and
optimizing alignment quality and fail to achieve fine-grained alignment due to
speech-text length mismatch. We introduce BLSP-KD, a novel approach for
Bootstrapping Language-Speech Pretraining via Knowledge Distillation, which
addresses these limitations through two key techniques. First, it optimizes
speech-text alignment by minimizing the divergence between the LLM's next-token
prediction distributions for speech and text inputs using knowledge
distillation. Second, it employs a continuous-integrate-andfire strategy to
segment speech into tokens that correspond one-to-one with text tokens,
enabling fine-grained alignment. We also introduce Partial LoRA (PLoRA), a new
adaptation method supporting LLM finetuning for speech inputs under knowledge
distillation. Quantitative evaluation shows that BLSP-KD outperforms previous
end-to-end baselines and cascaded systems with comparable scale of parameters,
facilitating general instruction-following capabilities for LLMs with speech
inputs. This approach provides new possibilities for extending LLMs to spoken
language interactions.

通过知识蒸馏，BLSP-KD 通过两个关键技术来优化语音 - 文本对齐质量，实现细粒度对齐，同时还引入了 LLM 的适应方法 PLoRA，通过定量评估说明了 BLSP-KD 在扩展 LLMs 到口语交互方面的优势。