The emergence of generative Large Language Models (LLMs) emphasizes the need
for accurate and efficient prompting approaches. LLMs are often applied in
Few-Shot Learning (FSL) contexts, where tasks are executed with minimal
training data. FSL has become popular in many Artificial Intelligence (AI)
subdomains, including AI for health. Rare diseases, affecting a small fraction
of the population, inherently require FSL techniques due to limited data
availability, though manual data collection and annotation is costly and
time-consuming. In this paper, we propose Models-Vote Prompting (MVP), a
flexible prompting approach for improving the performance of LLM queries in FSL
settings. MVP works by prompting numerous LLMs to perform the same tasks and
then conducting a majority vote on the resulting outputs. This method achieves
improved results to any one model in the ensemble on one-shot rare disease
identification and classification tasks. We also release a novel rare disease
dataset for FSL, available to those who agreed to the MIMIC-IV Data Use
Agreement (DUA). Furthermore, in using MVP, each model is prompted multiple
times, substantially increasing the time needed for manual annotation, and to
address this, we assess the feasibility of using JSON for automating generative
LLM evaluation.

我们提出了一种灵活的促进方法 ——Models-Vote Prompting (MVP)，以提高在 Few-Shot Learning (FSL) 环境中 LLM 查询的性能，并通过使用 MVP，在一次性罕见疾病的识别和分类任务上得到了比单个模型更好的结果。我们还发布了一份新颖的罕见疾病数据集，用于 FSL，并通过评估使用 JSON 来自动化生成性 LLM 评估的可行性来解决手动注释所需的时间问题。