The increasing use of tools and solutions based on Large Language Models
(LLMs) for various tasks in the medical domain has become a prominent trend.
Their use in this highly critical and sensitive domain has thus raised
important questions about their robustness, especially in response to
variations in input, and the reliability of the generated outputs. This study
addresses these questions by constructing a textual dataset based on the
ICD-10-CM code descriptions, widely used in US hospitals and containing many
clinical terms, and their easily reproducible rephrasing. We then benchmarked
existing embedding models, either generalist or specialized in the clinical
domain, in a semantic search task where the goal was to correctly match the
rephrased text to the original description. Our results showed that generalist
models performed better than clinical models, suggesting that existing clinical
specialized models are more sensitive to small changes in input that confuse
them. The highlighted problem of specialized models may be due to the fact that
they have not been trained on sufficient data, and in particular on datasets
that are not diverse enough to have a reliable global language understanding,
which is still necessary for accurate handling of medical documents.

使用大型语言模型（LLMs）进行医疗领域各种任务的工具和解决方案的使用日益增多，然而在这个高度关键和敏感的领域中，它们的稳健性以及所生成结果的可信度已经引起重要的问题。本研究通过构建基于 ICD-10-CM 代码描述的文本数据集来回答这些问题，该代码广泛应用于美国医院并包含许多临床术语及其易于复述。然后，我们对现有的嵌入模型进行了基准测试，无论是专门针对临床领域的模型还是通用模型，在一个语义搜索任务中，目标是将复述文本正确匹配到原始描述。我们的结果表明，通用模型的表现优于临床模型，这表明现有的临床专门模型更加敏感于输入的微小变化，从而使它们感到困惑。临床专门模型的突出问题可能是因为它们在训练时数据不够充分，特别是在没有足够多样化的数据集进行全局语言理解的情况下，而这对于准确处理医疗文档仍然是必要的。