Although Large Language Models (LLMs) have shown promise for human-like
conversations, they are primarily pre-trained on text data. Incorporating audio
or video improves performance, but collecting large-scale multimodal data and
pre-training multimodal LLMs is challenging. To this end, we propose a Fusion
Low Rank Adaptation (FLoRA) technique that efficiently adapts a pre-trained
unimodal LLM to consume new, previously unseen modalities via low rank
adaptation. For device-directed speech detection, using FLoRA, the multimodal
LLM achieves 22% relative reduction in equal error rate (EER) over the
text-only approach and attains performance parity with its full fine-tuning
(FFT) counterpart while needing to tune only a fraction of its parameters.
Furthermore, with the newly introduced adapter dropout, FLoRA is robust to
missing data, improving over FFT by 20% lower EER and 56% lower false accept
rate. The proposed approach scales well for model sizes from 16M to 3B
parameters.

通过使用 FLoRA 技术，将预训练的单模态大型语言模型适应到新的未见模态上，可以有效地改进多模态语言模型在设备导向的语音检测上的性能，并且通过引入适配器丢弃机制，提高了鲁棒性和模型的可扩展性。

多模式大型语言模型与融合低秩适应用于设备定向语音检测

Multimodal Large Language Models with Fusion Low Rank Adaptation for  Device Directed Speech Detection

Adapting pre-trained foundation models for various downstream tasks has been
prevalent in artificial intelligence. Due to the vast number of tasks and high
costs, adjusting all parameters becomes unfeasible. To mitigate this, several
fine-tuning techniques have been developed to update the pre-trained model
weights in a more resource-efficient manner, such as through low-rank
adjustments. Yet, almost all of these methods focus on linear weights,
neglecting the intricacies of parameter spaces in higher dimensions like 4D.
Alternatively, some methods can be adapted for high-dimensional parameter space
by compressing changes in the original space into two dimensions and then
employing low-rank matrix decomposition. However, these approaches destructs
the structural integrity of the involved high-dimensional spaces. To tackle the
diversity of dimensional spaces across different foundation models and provide
a more precise representation of the changes within these spaces, this paper
introduces a generalized parameter-efficient fine-tuning framework, FLoRA,
designed for various dimensional parameter space. Specifically, utilizing
Tucker decomposition, FLoRA asserts that changes in each dimensional parameter
space are based on a low-rank core space which maintains the consistent
topological structure with the original space. It then models the changes
through this core space alongside corresponding weights to reconstruct
alterations in the original space. FLoRA effectively preserves the structural
integrity of the change of original N-dimensional parameter space, meanwhile
decomposes it via low-rank tensor decomposition. Extensive experiments on
computer vision, natural language processing and multi-modal tasks validate
FLoRA's effectiveness. Codes are available at
this https URL

本文介绍了针对不同维参数空间的一种广义参数高效微调框架 FLoRA，通过 Tucker 分解在保持与原始空间一致的拓扑结构的低秩核空间上建模变化，并将相关权重用于重构原始空间中的变化，从而有效地保留原始 N 维参数空间变化的结构完整性，同时通过低秩张量分解对其进行分解。大量在计算机视觉、自然语言处理和多模态任务上的实验验证了 FLoRA 的有效性。

FLoRA: N 维低秩核空间

FLoRA: Low-Rank Core Space for N-dimension

Despite large neural networks demonstrating remarkable abilities to complete
different tasks, they require excessive memory usage to store the optimization
states for training. To alleviate this, the low-rank adaptation (LoRA) is
proposed to reduce the optimization states by training fewer parameters.
However, LoRA restricts overall weight update matrices to be low-rank, limiting
the model performance. In this work, we investigate the dynamics of LoRA and
identify that it can be approximated by a random projection. Based on this
observation, we propose Flora, which is able to achieve high-rank updates by
resampling the projection matrices while enjoying the sublinear space
complexity of optimization states. We conduct experiments across different
tasks and model architectures to verify the effectiveness of our approach.

提出了低秩适应（LoRA）方法，通过训练较少的参数来减少优化状态的存储消耗，但会限制整体权重更新矩阵为低秩，本研究基于观察到低秩适应可以用随机投影来近似，因此提出了 Flora 方法，通过重新采样投影矩阵实现高秩更新，同时享受优化状态的次线性空间复杂度，实验证明了我们方法的有效性。