The temperature parameter plays a profound role during training and/or
inference with large foundation models (LFMs) such as large language models
(LLMs) and CLIP models. Particularly, it adjusts the logits in the softmax
function in LLMs, which is crucial for next token generation, and it scales the
similarities in the contrastive loss for training CLIP models. A significant
question remains: Is it viable to learn a neural network to predict a
personalized temperature of any input data for enhancing LFMs"? In this paper,
we present a principled framework for learning a small yet generalizable
temperature prediction network (TempNet) to improve LFMs. Our solution is
composed of a novel learning framework with a robust loss underpinned by
constrained distributionally robust optimization (DRO), and a properly designed
TempNet with theoretical inspiration. TempNet can be trained together with a
large foundation model from scratch or learned separately given a pretrained
foundation model. It is not only useful for predicting personalized temperature
to promote the training of LFMs but also generalizable and transferable to new
tasks. Our experiments on LLMs and CLIP models demonstrate that TempNet greatly
improves the performance of existing solutions or models, e.g. Table 1. The
code to reproduce the experimental results in this paper can be found at
this https URL

本文提出了一个基于约束分布稳健优化（DRO）的有稳健性损失下的小而通用的温度预测网络（TempNet）的学习框架，并给出了相应的温度参数的定义和训练方法，实验证明 TempNet 可以显著改善现有解决方案或模型的性能。