Parameter-efficient fine-tuning (PEFT) methods have provided an effective way
for adapting large vision-language models to specific tasks or scenarios.
Typically, they learn a very small scale of parameters for pre-trained models
in a white-box formulation, which assumes model architectures to be known and
parameters to be accessible. However, large models are often not open-source
due to considerations of preventing abuse or commercial factors, hence posing a
barrier to the deployment of white-box PEFT methods. To alleviate the
dependence on model accessibility, we introduce collaborative black-box tuning
(CBBT) for both textual prompt optimization and output feature adaptation for
black-box models. Specifically, considering that the backpropagation gradients
are blocked, we approximate the gradients of textual prompts by analyzing the
predictions with perturbed prompts. Secondly, a lightweight adapter is deployed
over the output feature of the inaccessible model, further facilitating the
model adaptation process. Empowered with these designs, our CBBT is extensively
evaluated on eleven downstream benchmarks and achieves remarkable improvements
compared to existing black-box VL adaptation methods. Code is released at
this https URL

通过合作式黑盒调整，本研究介绍一种参数高效微调方法，用于对视觉语言模型进行特定任务或场景的适应。使用文本提示优化和输出特征调整来处理闭源模型的访问障碍，并在十一个下游基准测试中得到显著改进。