Black-box tuning has attracted recent attention due to that the structure or
inner parameters of advanced proprietary models are not accessible.
Proxy-tuning provides a test-time output adjustment for tuning black-box
language models. It applies the difference of the output logits before and
after tuning a smaller white-box "proxy" model to improve the black-box model.
However, this technique serves only as a decoding-time algorithm, leading to an
inconsistency between training and testing which potentially limits overall
performance. To address this problem, we introduce Consistent Proxy Tuning
(CPT), a simple yet effective black-box tuning method. Different from
Proxy-tuning, CPT additionally exploits the frozen large black-box model and
another frozen small white-box model, ensuring consistency between
training-stage optimization objective and test-time proxies. This consistency
benefits Proxy-tuning and enhances model performance. Note that our method
focuses solely on logit-level computation, which makes it model-agnostic and
applicable to any task involving logit classification. Extensive experimental
results demonstrate the superiority of our CPT in both black-box tuning of
Large Language Models (LLMs) and Vision-Language Models (VLMs) across various
datasets. The code is available at this https URL

黑盒调优方法中的一种，通过使用冻结的大型黑盒模型和另一个冻结的小型白盒模型确保一致性，以改进任何涉及逻辑分类任务的模型性能。

CPT：黑盒优化的一致代理调谐

CPT: Consistent Proxy Tuning for Black-box Optimization

Prompt-tuning has emerged as a promising method for adapting pre-trained
models to downstream tasks or aligning with human preferences. Prompt learning
is widely used in NLP but has limited applicability to RL due to the complex
physical meaning and environment-specific information contained within RL
prompts. These factors require supervised learning to imitate the
demonstrations and may result in a loss of meaning after learning.
Additionally, directly extending prompt-tuning approaches to RL is challenging
because RL prompts guide agent behavior based on environmental modeling and
analysis, rather than filling in missing information, making it unlikely that
adjustments to the prompt format for downstream tasks, as in NLP, can yield
significant improvements. In this work, we propose the Prompt-Tuning DT
algorithm to address these challenges by using trajectory segments as prompts
to guide RL agents in acquiring environmental information and optimizing
prompts via black-box tuning to enhance their ability to contain more relevant
information, thereby enabling agents to make better decisions. Our approach
involves randomly sampling a Gaussian distribution to fine-tune the elements of
the prompt trajectory and using preference ranking function to find the
optimization direction, thereby providing more informative prompts and guiding
the agent towards specific preferences in the target environment. Extensive
experiments show that with only 0.03% of the parameters learned, Prompt-Tuning
DT achieves comparable or even better performance than full-model fine-tuning
in low-data scenarios. Our work contributes to the advancement of prompt-tuning
approaches in RL, providing a promising direction for optimizing large RL
agents for specific preference tasks.

本文提出了 Prompt-Tuning DT 算法，使用轨迹段作为提示来指导强化学习（RL）代理获取环境信息并通过黑盒调整来优化提示，以提供更多相关信息和指导代理走向特定任务的方向，在低数据情况下仅学习 0.03％的参数即可实现与全模型微调相当甚至更好的性能，为 RL 中优化大型代理的特定任务提供了有前途的方向。

具备偏好排名的 Prompt-Tuning 决策 Transformer

Prompt-Tuning Decision Transformer with Preference Ranking

Large language models (LLMs) have shown increasing power on various natural
language processing (NLP) tasks. However, tuning these models for downstream
tasks usually needs exorbitant costs or is unavailable due to commercial
considerations. Recently, black-box tuning has been proposed to address this
problem by optimizing task-specific prompts without accessing the gradients and
hidden representations. However, most existing works have yet fully exploited
the potential of gradient-free optimization under the scenario of few-shot
learning. In this paper, we describe BBT-RGB, a suite of straightforward and
complementary techniques for enhancing the efficiency and performance of
black-box optimization. Specifically, our method includes three plug-and-play
components: (1) Two-stage derivative-free optimization strategy that
facilitates fast convergence and mitigates overfitting; (2) Automatic
verbalizer construction with its novel usage under few-shot settings; (3)
Better prompt initialization policy based on instruction search and
auto-selected demonstration. Extensive experiments across various tasks on
natural language understanding and inference demonstrate the effectiveness of
our method. Our codes are publicly available at
this https URL

本文提出 BBT-RGB, 一套用于提高黑盒优化效率和性能的简单而相辅相成的技术。我们的实验结果表明，该方法在各种自然语言理解和推理任务中具有很好的有效性。

基于提示的黑盒调参：从三个正交视角提高模型的泛化能力

Make Prompt-based Black-Box Tuning Colorful: Boosting Model  Generalization from Three Orthogonal Perspectives

Prompt tuning recently becomes a hot-spot in the applications of large
pretrained language models on specific downstream tasks. Regarding the Language
Model as a Service (LMaaS), black-box tuning using derivative-free optimization
(DFO) provides a novel approach to expand the practical scenarios of pretrained
models and enrich the researches of few-shot learning. In this report, we
present our solution in this competition that is based on the LMaaS scenario.
Our solution consists of several modifications to BBTv2, including multiple
label words, selection of P0, rolling update strategy, multi-task loss from MLP
classifier, and finally using the ensemble method to further improve
generalization ability. We also shared some strategies that we tried but didn't
use in the final submission for further discussion. In the end we raised a
question about the SNLI dataset and the impact on the results, as well as our
concerns about the competition.

本报告是关于在预训练语言模型中应用黑盒调整以进行 few-shot 学习的研究。 我们提出了基于 LMaaS 场景的解决方案，包括多标签词、P0 选择、滚动更新策略、MLP 分类器的多任务损失以及使用集成方法进一步提高广义化能力。 最后，我们提出了有关 SNLI 数据集对结果影响的问题，以及我们对比赛的担忧。