The success of Large Vision Models (LVMs) is accompanied by vast data
volumes, which are prohibitively expensive in medical diagnosis.To address
this, recent efforts exploit Parameter-Efficient Fine-Tuning (PEFT), which
trains a small number of weights while freezing the rest.However, they
typically assign trainable weights to the same positions in LVMs in a heuristic
manner, regardless of task differences, making them suboptimal for professional
applications like medical diagnosis.To address this, we statistically reveal
the nature of sparsity and hybridity during diagnostic-targeted fine-tuning,
i.e., a small portion of key weights significantly impacts performance, and
these key weights are hybrid, including both task-specific and task-agnostic
parts.Based on this, we propose a novel Sparsity- and Hybridity-inspired
Parameter Efficient Fine-Tuning (SH-PEFT).It selects and trains a small portion
of weights based on their importance, which is innovatively estimated by
hybridizing both task-specific and task-agnostic strategies.Validated on six
medical datasets of different modalities, we demonstrate that SH-PEFT achieves
state-of-the-art performance in transferring LVMs to medical diagnosis in terms
of accuracy. By tuning around 0.01% number of weights, it outperforms full
model fine-tuning.Moreover, SH-PEFT also achieves comparable performance to
other models deliberately optimized for specific medical tasks.Extensive
experiments demonstrate the effectiveness of each design and reveal that large
model transfer holds great potential in medical diagnosis.

基于 LVM 的 Sparsity 和 Hybridity 特性，利用 SH-PEFT 方法在医学诊断中通过训练少量的权重，超越了全模型微调，并达到了同其他针对特定医学任务优化的模型可比较的性能，表明大型模型传输在医学诊断中具有巨大潜力。