Large language models (LLMs) have shown remarkable performance on many tasks
in different domains. However, their performance in closed-book biomedical
machine reading comprehension (MRC) has not been evaluated in depth. In this
work, we evaluate GPT on four closed-book biomedical MRC benchmarks. We
experiment with different conventional prompting techniques as well as
introduce our own novel prompting method. To solve some of the retrieval
problems inherent to LLMs, we propose a prompting strategy named Implicit
Retrieval Augmented Generation (RAG) that alleviates the need for using vector
databases to retrieve important chunks in traditional RAG setups. Moreover, we
report qualitative assessments on the natural language generation outputs from
our approach. The results show that our new prompting technique is able to get
the best performance in two out of four datasets and ranks second in rest of
them. Experiments show that modern-day LLMs like GPT even in a zero-shot
setting can outperform supervised models, leading to new state-of-the-art
(SoTA) results on two of the benchmarks.

我们评估了 GPT 在四个封闭式生物医学机器阅读理解基准测试上的表现，提出了一种名为 Implicit Retrieval Augmented Generation（IRAG）的提示策略，该策略通过减少传统 RAG 设置中使用向量数据库检索重要部分的需求来解决 LLM 所固有的检索问题，并通过定性评估展示了该方法的自然语言生成输出。实验结果表明，我们的新提示技术在四个数据集中有两个取得了最佳效果，并在其余两个中排名第二。实验还表明，像 GPT 这样的现代 LLM，即使在零 - shot 设置中，也能胜过监督模型，从而在两个基准测试中取得了最新技术水平的成果。

GPT 能否重新定义医学认识？对生物医学机器阅读理解中的 GPT 进行评估

Can GPT Redefine Medical Understanding? Evaluating GPT on Biomedical  Machine Reading Comprehension

Biomedical machine reading comprehension (biomedical-MRC) aims to comprehend
complex biomedical narratives and assist healthcare professionals in retrieving
information from them. The high performance of modern neural network-based MRC
systems depends on high-quality, large-scale, human-annotated training
datasets. In the biomedical domain, a crucial challenge in creating such
datasets is the requirement for domain knowledge, inducing the scarcity of
labeled data and the need for transfer learning from the labeled
general-purpose (source) domain to the biomedical (target) domain. However,
there is a discrepancy in marginal distributions between the general-purpose
and biomedical domains due to the variances in topics. Therefore,
direct-transferring of learned representations from a model trained on a
general-purpose domain to the biomedical domain can hurt the model's
performance. We present an adversarial learning-based domain adaptation
framework for the biomedical machine reading comprehension task (BioADAPT-MRC),
a neural network-based method to address the discrepancies in the marginal
distributions between the general and biomedical domain datasets. BioADAPT-MRC
relaxes the need for generating pseudo labels for training a well-performing
biomedical-MRC model. We extensively evaluate the performance of BioADAPT-MRC
by comparing it with the best existing methods on three widely used benchmark
biomedical-MRC datasets -- BioASQ-7b, BioASQ-8b, and BioASQ-9b. Our results
suggest that without using any synthetic or human-annotated data from the
biomedical domain, BioADAPT-MRC can achieve state-of-the-art performance on
these datasets. Availability: BioADAPT-MRC is freely available as an
open-source project at https://github.com/mmahbub/BioADAPT-MRC.

本篇论文提出了一种基于对抗学习的领域适应框架，即 BioADAPT-MRC，用于解决通用域和生物医学域数据集之间的分布分歧问题，使得在不使用合成或人工注释的数据的情况下，可以实现在生物医学文本阅读理解任务中最好的性能。