When deploying machine learning models in production for any
product/application, there are three properties that are commonly desired.
First, the models should be generalizable, in that we can extend it to further
use cases as our knowledge of the domain area develops. Second they should be
evaluable, so that there are clear metrics for performance and the calculation
of those metrics in production settings are feasible. Finally, the deployment
should be cost-optimal as far as possible. In this paper we propose that these
three objectives (i.e. generalization, evaluation and cost-optimality) can
often be relatively orthogonal and that for large language models, despite
their performance over conventional NLP models, enterprises need to carefully
assess all the three factors before making substantial investments in this
technology. We propose a framework for generalization, evaluation and
cost-modeling specifically tailored to large language models, offering insights
into the intricacies of development, deployment and management for these large
language models.

通过提出一个专为大型语言模型量身定制的泛化、评估和成本建模框架，我们认为在部署大型语言模型之前，企业需要仔细评估泛化、评估和成本优化三个因素。