Generative language models (LMs) have become omnipresent across data science.
For a wide variety of tasks, inputs can be phrased as natural language prompts
for an LM, from whose output the solution can then be extracted. LM performance
has consistently been increasing with model size - but so has the monetary cost
of querying the ever larger models. Importantly, however, not all inputs are
equally hard: some require larger LMs for obtaining a satisfactory solution,
whereas for others smaller LMs suffice. Based on this fact, we design a
framework for Cost-Effective Language Model Choice (CELMOC). Given a set of
inputs and a set of candidate LMs, CELMOC judiciously assigns each input to an
LM predicted to do well on the input according to a so-called meta-model,
aiming to achieve high overall performance at low cost. The cost-performance
trade-off can be flexibly tuned by the user. Options include, among others,
maximizing total expected performance (or the number of processed inputs) while
staying within a given cost budget, or minimizing total cost while processing
all inputs. We evaluate CELMOC on 14 datasets covering five natural language
tasks, using four candidate LMs of vastly different size and cost. With CELMOC,
we match the performance of the largest available LM while achieving a cost
reduction of 63%. Via our publicly available library, researchers as well as
practitioners can thus save large amounts of money without sacrificing
performance.

基于句子中给定的输入和候选语言模型，CELMOC 框架致力于以低成本实现高整体性能，最大程度地节省经费并匹配最大可用的语言模型的性能。