In our increasingly interconnected world, where intelligent devices
continually amass copious personalized multi-modal data, a pressing need arises
to deliver high-quality, personalized device-aware services. However, this
endeavor presents a multifaceted challenge to prevailing artificial
intelligence (AI) systems primarily rooted in the cloud. As these systems
grapple with shifting data distributions between the cloud and devices, the
traditional approach of fine-tuning-based adaptation (FTA) exists the following
issues: the costly and time-consuming data annotation required by FTA and the
looming risk of model overfitting. To surmount these challenges, we introduce a
Universal On-Device Multi-modal Model Adaptation Framework, revolutionizing
on-device model adaptation by striking a balance between efficiency and
effectiveness. The framework features the Fast Domain Adaptor (FDA) hosted in
the cloud, providing tailored parameters for the Lightweight Multi-modal Model
on devices. To enhance adaptability across multi-modal tasks, the AnchorFrame
Distribution Reasoner (ADR) minimizes communication costs. Our contributions,
encapsulated in the Cloud-Device Collaboration Multi-modal Parameter Generation
(CDC-MMPG) framework, represent a pioneering solution for on-Device Multi-modal
Model Adaptation (DMMA). Extensive experiments validate the efficiency and
effectiveness of our method, particularly in video question answering and
retrieval tasks, driving forward the integration of intelligent devices into
our daily lives.

我们在日益相互连接的世界中，智能设备不断积累大量的个性化多模态数据，迫切需要提供高质量、个性化的设备感知服务。为了克服云端基于微调适应 (FTA) 存在的问题，我们引入了一种革新的通用 On-Device 多模态模型适应框架，通过在云端部署 Fast Domain Adaptor (FDA) 和设备上的轻量级多模态模型，实现了效率和效果的平衡。为了增强跨多模态任务的适应性，AnchorFrame Distribution Reasoner (ADR) 减少了通信成本。我们的贡献体现在 Cloud-Device Collaboration 多模态参数生成 (CDC-MMPG) 框架中，为 On-Device 多模态模型适应 (DMMA) 提供了先驱性的解决方案。广泛的实验证实了我们的方法的效率和效果，特别是在视频问答和检索任务中，推动了智能设备融入我们的日常生活。