The recent success of large language models (LLMs) has paved the way for
their adoption in the high-stakes domain of healthcare. Specifically, the
application of LLMs in patient-trial matching, which involves assessing patient
eligibility against clinical trial's nuanced inclusion and exclusion criteria,
has shown promise. Recent research has shown that GPT-3.5, a widely recognized
LLM developed by OpenAI, can outperform existing methods with minimal 'variable
engineering' by simply comparing clinical trial information against patient
summaries. However, there are significant challenges associated with using
closed-source proprietary LLMs like GPT-3.5 in practical healthcare
applications, such as cost, privacy and reproducibility concerns. To address
these issues, this study presents the first systematic examination of the
efficacy of both proprietary (GPT-3.5, and GPT-4) and open-source LLMs (LLAMA
7B,13B, and 70B) for the task of patient-trial matching. Employing a
multifaceted evaluation framework, we conducted extensive automated and
human-centric assessments coupled with a detailed error analysis for each
model. To enhance the adaptability of open-source LLMs, we have created a
specialized synthetic dataset utilizing GPT-4, enabling effective fine-tuning
under constrained data conditions. Our findings reveal that open-source LLMs,
when fine-tuned on this limited and synthetic dataset, demonstrate performance
parity with their proprietary counterparts. This presents a massive opportunity
for their deployment in real-world healthcare applications. To foster further
research and applications in this field, we release both the annotated
evaluation dataset along with the fine-tuned LLM -- Trial-LLAMA -- for public
use.

探究专有和开源大型语言模型在患者试验匹配任务中的有效性，并证明在有限和合成数据集上进行微调后，开源模型与专有模型性能相当，从而为它们在实际医疗应用中的部署提供了巨大机会。

将大型语言模型凝聚用于将患者匹配到临床试验

Distilling Large Language Models for Matching Patients to Clinical  Trials

Recent observations have underscored a disparity between the inflated
benchmark scores and the actual performance of LLMs, raising concerns about
potential contamination of evaluation benchmarks. This issue is especially
critical for closed-source models and certain open-source models where training
data transparency is lacking. In this paper we study data contamination by
proposing two methods tailored for both open-source and proprietary LLMs. We
first introduce a retrieval-based system to explore potential overlaps between
evaluation benchmarks and pretraining corpora. We further present a novel
investigation protocol named \textbf{T}estset \textbf{S}lot Guessing
(\textit{TS-Guessing}), applicable to both open and proprietary models. This
approach entails masking a wrong answer in a multiple-choice question and
prompting the model to fill in the gap. Additionally, it involves obscuring an
unlikely word in an evaluation example and asking the model to produce it. We
find that certain commercial LLMs could surprisingly guess the missing option
in various test sets. Specifically, in the TruthfulQA benchmark, we find that
LLMs exhibit notable performance improvement when provided with additional
metadata in the benchmark. Further, in the MMLU benchmark, ChatGPT and GPT-4
demonstrated an exact match rate of 52\% and 57\%, respectively, in guessing
the missing options in benchmark test data. We hope these results underscore
the need for more robust evaluation methodologies and benchmarks in the field.

我们通过检测数据污染并提出 “Testset Slot Guessing” 方法，发现商业化的 LLM 在评估基准中有明显的性能改进，并呼吁在该领域使用更强大的评估方法和基准。

现代大语言模型中数据污染问题研究

Investigating Data Contamination in Modern Benchmarks for Large Language  Models

The dominance of proprietary LLMs has led to restricted access and raised
information privacy concerns. High-performing open-source alternatives are
crucial for information-sensitive and high-volume applications but often lag
behind in performance. To address this gap, we propose (1) A untargeted variant
of iterative self-critique and self-refinement devoid of external influence.
(2) A novel ranking metric - Performance, Refinement, and Inference Cost Score
(PeRFICS) - to find the optimal model for a given task considering refined
performance and cost. Our experiments show that SoTA open source models of
varying sizes from 7B - 65B, on average, improve 8.2% from their baseline
performance. Strikingly, even models with extremely small memory footprints,
such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39%
improvement in high-creativity, open ended tasks on the Vicuna benchmark.
Vicuna-13B takes it a step further and outperforms ChatGPT post-refinement.
This work has profound implications for resource-constrained and
information-sensitive environments seeking to leverage LLMs without incurring
prohibitive costs, compromising on performance and privacy. The domain-agnostic
self-refinement process coupled with our novel ranking metric facilitates
informed decision-making in model selection, thereby reducing costs and
democratizing access to high-performing language models, as evidenced by case
studies.

我们提出了一种无外部干扰的非定向迭代自我批判和自我完善的方法，以及一种新颖的评分指标，用于在考虑到性能和成本的情况下为给定任务找到最佳模型。我们的实验表明，规模不同的 SoTA 开源模型在性能方面平均提高 8.2%。这对希望在不牺牲性能和隐私的情况下利用 LLMs 的资源有限和信息敏感环境具有重要意义。