The emerging trend of advancing generalist artificial intelligence, such as
GPTv4 and Gemini, has reshaped the landscape of research (academia and
industry) in machine learning and many other research areas. However,
domain-specific applications of such foundation models (e.g., in medicine)
remain untouched or often at their very early stages. It will require an
individual set of transfer learning and model adaptation techniques by further
expanding and injecting these models with domain knowledge and data. The
development of such technologies could be largely accelerated if the bundle of
data, algorithms, and pre-trained foundation models were gathered together and
open-sourced in an organized manner. In this work, we present OpenMEDLab, an
open-source platform for multi-modality foundation models. It encapsulates not
only solutions of pioneering attempts in prompting and fine-tuning large
language and vision models for frontline clinical and bioinformatic
applications but also building domain-specific foundation models with
large-scale multi-modal medical data. Importantly, it opens access to a group
of pre-trained foundation models for various medical image modalities, clinical
text, protein engineering, etc. Inspiring and competitive results are also
demonstrated for each collected approach and model in a variety of benchmarks
for downstream tasks. We welcome researchers in the field of medical artificial
intelligence to continuously contribute cutting-edge methods and models to
OpenMEDLab, which can be accessed via this https URL

OpenMEDLab 是一个开放源代码平台，用于多模态基础模型，它不仅集成了前沿临床和生物信息学应用的大型语言和视觉模型的解决方案，还利用大规模多模态医学数据构建领域特定的基础模型。该平台还提供了各种医学图像模态、临床文本和蛋白工程等领域的预训练基础模型，并在各种基准测试中展示了激动人心的竞争结果。欢迎医学人工智能领域的研究人员不断为 OpenMEDLab 贡献前沿的方法和模型。

OpenMEDLab：一个面向医学领域的开源多模态基础模型平台

OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models  in Medicine

Sub-population shift is a specific type of domain shift that highlights
changes in data distribution within specific sub-groups or populations between
training and testing. Sub-population shift accounts for a significant source of
algorithmic bias and calls for distributional robustness. Recent studies found
inherent distributional robustness in multi-modality foundation models, such as
the vision-language model CLIP, yet this robustness is vulnerable through
parameter fine-tuning. In this paper, we propose leveraging the connection of
robustness among different modalities and reshaping the distributional
robustness of one modality with another. Specifically, in the context of the
distributional robustness of CLIP, we propose to leverage natural language
inputs to debias the image feature representations, to improve worst-case
performance on sub-populations. Our extensive empirical studies show that image
representations debiased by natural language can achieve significant
performance improvement and reduction of performance instability under
sub-population shifts.

本文通过利用多模态基础模型中的分布鲁棒性，通过参数微调来提高分布鲁棒性，并利用自然语言输入来消除图像特征中的偏见，从而在子群落变化下改善最坏情况性能。大量实证研究表明，通过自然语言进行去偏倚的图像表示能够显著提高性能，并降低在子群体变化下的性能不稳定性。

跨模态去偏倚：利用语言减轻图像中的亚种群转变

Cross-modality debiasing: using language to mitigate sub-population  shifts in imaging

Multi-modality foundation models, as represented by GPT-4V, have brought a
new paradigm for low-level visual perception and understanding tasks, that can
respond to a broad range of natural human instructions in a model. While
existing foundation models have shown exciting potentials on low-level visual
tasks, their related abilities are still preliminary and need to be improved.
In order to enhance these models, we conduct a large-scale subjective
experiment collecting a vast number of real human feedbacks on low-level
vision. Each feedback follows a pathway that starts with a detailed description
on the low-level visual appearance (*e.g. clarity, color, brightness* of an
image, and ends with an overall conclusion, with an average length of 45 words.
The constructed **Q-Pathway** dataset includes 58K detailed human feedbacks on
18,973 images with diverse low-level appearance. Moreover, to enable foundation
models to robustly respond to diverse types of questions, we design a
GPT-participated conversion to process these feedbacks into diverse-format 200K
instruction-response pairs. Experimental results indicate that the
**Q-Instruct** consistently elevates low-level perception and understanding
abilities across several foundational models. We anticipate that our datasets
can pave the way for a future that general intelligence can perceive,
understand low-level visual appearance and evaluate visual quality like a
human. Our dataset, model zoo, and demo is published at:
this https URL

基于 GPT-4V 的多模态基础模型，在低级视觉感知和理解任务方面带来了新的范式，可以对多种自然人类指令做出响应。通过大规模的主观实验收集了大量关于低级视觉的真实人类反馈，建立了包含 58K 个详细反馈的 Q-Pathway 数据集，实验结果表明，Q-Instruct 能够提升多个基础模型在低级感知和理解能力方面的表现，我们的数据集和模型展示可在所发布的网址获取。