Recent advancements in multimodal foundation models have showcased impressive
capabilities in understanding and reasoning with visual and textual
information. Adapting these foundation models trained for general usage to
specialized domains like biomedicine requires large-scale domain-specific
instruction datasets. While existing works have explored curating such datasets
automatically, the resultant datasets are not explicitly aligned with domain
expertise. In this work, we propose a data-centric framework, Biomedical Visual
Instruction Tuning with Clinician Preference Alignment (BioMed-VITAL), that
incorporates clinician preferences into both stages of generating and selecting
instruction data for tuning biomedical multimodal foundation models. First,
during the generation stage, we prompt the GPT-4V generator with a diverse set
of clinician-selected demonstrations for preference-aligned data candidate
generation. Then, during the selection phase, we train a separate selection
model, which explicitly distills clinician and policy-guided model preferences
into a rating function to select high-quality data for medical instruction
tuning. Results show that the model tuned with the instruction-following data
from our method demonstrates a significant improvement in open visual chat
(18.5% relatively) and medical VQA (win rate up to 81.73%). Our
instruction-following data and models are available at BioMed-VITAL.github.io.

本文提出了一种数据中心的框架 BioMed-VITAL，将临床医生的偏好纳入到生成和选择生物医学多模式基础模型调整的指导数据的过程中，并通过这种方法获得的指导数据进行调整后的模型在开放式视觉对话和医疗问答中表现出显著改进。

生物医学视觉教学优化与临床医师偏好一致性调整

Biomedical Visual Instruction Tuning with Clinician Preference Alignment

Continuous machine learning pipelines are common in industrial settings where
models are periodically trained on data streams. Unfortunately, concept drifts
may occur in data streams where the joint distribution of the data X and label
y, P(X, y), changes over time and possibly degrade model accuracy. Existing
concept drift adaptation approaches mostly focus on updating the model to the
new data possibly using ensemble techniques of previous models and tend to
discard the drifted historical data. However, we contend that explicitly
utilizing the drifted data together leads to much better model accuracy and
propose Quilt, a data-centric framework for identifying and selecting data
segments that maximize model accuracy. To address the potential downside of
efficiency, Quilt extends existing data subset selection techniques, which can
be used to reduce the training data without compromising model accuracy. These
techniques cannot be used as is because they only assume virtual drifts where
the posterior probabilities P(y|X) are assumed not to change. In contrast, a
key challenge in our setup is to also discard undesirable data segments with
concept drifts. Quilt thus discards drifted data segments and selects data
segment subsets holistically for accurate and efficient model training. The two
operations use gradient-based scores, which have little computation overhead.
In our experiments, we show that Quilt outperforms state-of-the-art drift
adaptation and data selection baselines on synthetic and real datasets.

通过使用 Quilt，这是一个数据中心的框架来鉴定和选择能够最大化模型准确性的数据片段，从而提高模型准确性和效率。

Quilt：面对概念漂移的鲁棒数据段选择

Quilt: Robust Data Segment Selection against Concept Drifts

Evaluating the value of a hypothetical target policy with only a logged
dataset is important but challenging. On the one hand, it brings opportunities
for safe policy improvement under high-stakes scenarios like clinical
guidelines. On the other hand, such opportunities raise a need for precise
off-policy evaluation (OPE). While previous work on OPE focused on improving
the algorithm in value estimation, in this work, we emphasize the importance of
the offline dataset, hence putting forward a data-centric framework for
evaluating OPE problems. We propose DataCOPE, a data-centric framework for
evaluating OPE, that answers the questions of whether and to what extent we can
evaluate a target policy given a dataset. DataCOPE (1) forecasts the overall
performance of OPE algorithms without access to the environment, which is
especially useful before real-world deployment where evaluating OPE is
impossible; (2) identifies the sub-group in the dataset where OPE can be
inaccurate; (3) permits evaluations of datasets or data-collection strategies
for OPE problems. Our empirical analysis of DataCOPE in the logged contextual
bandit settings using healthcare datasets confirms its ability to evaluate both
machine-learning and human expert policies like clinical guidelines.

通过数据集进行离线策略评估是重要但具有挑战性的，本研究提出了一种数据中心的框架用于评估其问题，并且通过对医疗数据集的实证分析验证了其能够评估机器学习和人工专家策略。