In the rapidly evolving landscape of Natural Language Processing (NLP), Large
Language Models (LLMs) have demonstrated remarkable capabilities in tasks such
as question answering (QA). However, the accessibility and practicality of
utilizing these models for industrial applications pose significant challenges,
particularly concerning cost-effectiveness, inference speed, and resource
efficiency. This paper presents a comprehensive benchmarking study comparing
open-source LLMs with their non-open-source counterparts on the task of
question answering. Our objective is to identify open-source alternatives
capable of delivering comparable performance to proprietary models while being
lightweight in terms of resource requirements and suitable for Central
Processing Unit (CPU)-based inference. Through rigorous evaluation across
various metrics including accuracy, inference speed, and resource consumption,
we aim to provide insights into selecting efficient LLMs for real-world
applications. Our findings shed light on viable open-source alternatives that
offer acceptable performance and efficiency, addressing the pressing need for
accessible and efficient NLP solutions in industry settings.

该论文通过对问题回答任务进行综合基准测试研究，旨在识别对实际应用具有可比较性能且资源要求轻的开源替代方案，以解决工业领域对可访问和高效的自然语言处理解决方案的紧迫需求。

工业应用中高效问答的开源语言模型基准测试

Benchmarking Open-Source Language Models for Efficient Question  Answering in Industrial Applications

The dominance of proprietary LLMs has led to restricted access and raised
information privacy concerns. High-performing open-source alternatives are
crucial for information-sensitive and high-volume applications but often lag
behind in performance. To address this gap, we propose (1) A untargeted variant
of iterative self-critique and self-refinement devoid of external influence.
(2) A novel ranking metric - Performance, Refinement, and Inference Cost Score
(PeRFICS) - to find the optimal model for a given task considering refined
performance and cost. Our experiments show that SoTA open source models of
varying sizes from 7B - 65B, on average, improve 8.2% from their baseline
performance. Strikingly, even models with extremely small memory footprints,
such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39%
improvement in high-creativity, open ended tasks on the Vicuna benchmark.
Vicuna-13B takes it a step further and outperforms ChatGPT post-refinement.
This work has profound implications for resource-constrained and
information-sensitive environments seeking to leverage LLMs without incurring
prohibitive costs, compromising on performance and privacy. The domain-agnostic
self-refinement process coupled with our novel ranking metric facilitates
informed decision-making in model selection, thereby reducing costs and
democratizing access to high-performing language models, as evidenced by case
studies.

我们提出了一种无外部干扰的非定向迭代自我批判和自我完善的方法，以及一种新颖的评分指标，用于在考虑到性能和成本的情况下为给定任务找到最佳模型。我们的实验表明，规模不同的 SoTA 开源模型在性能方面平均提高 8.2%。这对希望在不牺牲性能和隐私的情况下利用 LLMs 的资源有限和信息敏感环境具有重要意义。