The dominance of proprietary LLMs has led to restricted access and raised
information privacy concerns. High-performing open-source alternatives are
crucial for information-sensitive and high-volume applications but often lag
behind in performance. To address this gap, we propose (1) A untargeted variant
of iterative self-critique and self-refinement devoid of external influence.
(2) A novel ranking metric - Performance, Refinement, and Inference Cost Score
(PeRFICS) - to find the optimal model for a given task considering refined
performance and cost. Our experiments show that SoTA open source models of
varying sizes from 7B - 65B, on average, improve 8.2% from their baseline
performance. Strikingly, even models with extremely small memory footprints,
such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39%
improvement in high-creativity, open ended tasks on the Vicuna benchmark.
Vicuna-13B takes it a step further and outperforms ChatGPT post-refinement.
This work has profound implications for resource-constrained and
information-sensitive environments seeking to leverage LLMs without incurring
prohibitive costs, compromising on performance and privacy. The domain-agnostic
self-refinement process coupled with our novel ranking metric facilitates
informed decision-making in model selection, thereby reducing costs and
democratizing access to high-performing language models, as evidenced by case
studies.

我们提出了一种无外部干扰的非定向迭代自我批判和自我完善的方法，以及一种新颖的评分指标，用于在考虑到性能和成本的情况下为给定任务找到最佳模型。我们的实验表明，规模不同的 SoTA 开源模型在性能方面平均提高 8.2%。这对希望在不牺牲性能和隐私的情况下利用 LLMs 的资源有限和信息敏感环境具有重要意义。