Large Language Models have shown promising results in their ability to encode
general medical knowledge in standard medical question-answering datasets.
However, their potential application in clinical practice requires evaluation
in domain-specific tasks, where benchmarks are largely missing. In this study
semioLLM, we test the ability of state-of-the-art LLMs (GPT-3.5, GPT-4, Mixtral
8x7B, and Qwen-72chat) to leverage their internal knowledge and reasoning for
epilepsy diagnosis. Specifically, we obtain likelihood estimates linking
unstructured text descriptions of seizures to seizure-generating brain regions,
using an annotated clinical database containing 1269 entries. We evaluate the
LLM's performance, confidence, reasoning, and citation abilities in comparison
to clinical evaluation. Models achieve above-chance classification performance
with prompt engineering significantly improving their outcome, with some models
achieving close-to-clinical performance and reasoning. However, our analyses
also reveal significant pitfalls with several models being overly confident
while showing poor performance, as well as exhibiting citation errors and
hallucinations. In summary, our work provides the first extensive benchmark
comparing current SOTA LLMs in the medical domain of epilepsy and highlights
their ability to leverage unstructured texts from patients' medical history to
aid diagnostic processes in health care.

本研究通过对临床数据库中的文本描述进行测试，评估了最新的大型语言模型在癫痫诊断中结合内部知识和推理的能力，并提供了首个对当前 SOTA 大型语言模型在癫痫医学领域的综合评估结果，强调了它们利用患者病史中的非结构化文本辅助诊断过程的能力。

SemioLLM：大型语言模型在癫痫研究中的符号学分析评估

SemioLLM: Assessing Large Language Models for Semiological Analysis in  Epilepsy Research

MR images scanned at low magnetic field ($<1$T) have lower resolution in the
slice direction and lower contrast, due to a relatively small signal-to-noise
ratio (SNR) than those from high field (typically 1.5T and 3T). We adapt the
recent idea of Image Quality Transfer (IQT) to enhance very low-field
structural images aiming to estimate the resolution, spatial coverage, and
contrast of high-field images. Analogous to many learning-based image
enhancement techniques, IQT generates training data from high-field scans alone
by simulating low-field images through a pre-defined decimation model. However,
the ground truth decimation model is not well-known in practice, and lack of
its specification can bias the trained model, aggravating performance on the
real low-field scans. In this paper we propose a probabilistic decimation
simulator to improve robustness of model training. It is used to generate and
augment various low-field images whose parameters are random variables and
sampled from an empirical distribution related to tissue-specific SNR on a
0.36T scanner. The probabilistic decimation simulator is model-agnostic, that
is, it can be used with any super-resolution networks. Furthermore we propose a
variant of U-Net architecture to improve its learning performance. We show
promising qualitative results from clinical low-field images confirming the
strong efficacy of IQT in an important new application area: epilepsy diagnosis
in sub-Saharan Africa where only low-field scanners are normally available.

该研究采用图像质量转移（IQT）的方法，提高低场（<1T）扫描的结构 MRI 图像的分辨率、空间覆盖度和对比度，通过引入概率减采样模型，增强模型的鲁棒性，并设计了一种 U-net 架构的变种，在亚非地区低场 MRI 诊断癫痫的应用中取得了良好的效果。