Domain shift differences between training data for deep learning models and
the deployment context can result in severe performance issues for models which
fail to generalize. We study the domain shift problem under the context of
remote photoplethysmography (rPPG), a technique for video-based heart rate
inference. We propose metrics based on model similarity which may be used as a
measure of domain shift, and we demonstrate high correlation between these
metrics and empirical performance. One of the proposed metrics with viable
correlations, DS-diff, does not assume access to the ground truth of the target
domain, i.e. it may be applied to in-the-wild data. To that end, we investigate
a model selection problem in which ground truth results for the evaluation
domain is not known, demonstrating a 13.9% performance improvement over the
average case baseline.

通过模型相似性来衡量领域偏移的指标可用作领域偏移的度量，并且我们展示了这些指标与实际性能之间的高相关性。其中一项具有可行相关性的提议指标为 DS-diff，不需要假设对目标领域的基本事实，即可应用于自然环境数据。因此，我们研究了一个模型选择问题，在该问题中，无法获得评估领域的基本事实结果，并且相对于平均情况基准线，性能提高了 13.9%。

使用深度学习远程光电容积脉搏模型相似性测量领域转变

Measuring Domain Shifts using Deep Learning Remote Photoplethysmography  Model Similarity

Foundation models have recently expanded into robotics after excelling in
computer vision and natural language processing. The models are accessible in
two ways: open-source or paid, closed-source options. Users with access to both
face a problem when deciding between effective yet costly closed-source models
and free but less powerful open-source alternatives. We call it the model
selection problem. Existing supervised-learning methods are impractical due to
the high cost of collecting extensive training data from closed-source models.
Hence, we focus on the online learning setting where algorithms learn while
collecting data, eliminating the need for large pre-collected datasets. We thus
formulate a user-centric online model selection problem and propose a novel
solution that combines an open-source encoder to output context and an online
learning algorithm that processes this context. The encoder distills vast data
distributions into low-dimensional features, i.e., the context, without
additional training. The online learning algorithm aims to maximize a composite
reward that includes model performance, execution time, and costs based on the
context extracted from the data. It results in an improved trade-off between
selecting open-source and closed-source models compared to non-contextual
methods, as validated by our theoretical analysis. Experiments across
language-based robotic tasks such as Waymo Open Dataset, ALFRED, and Open
X-Embodiment demonstrate real-world applications of the solution. The results
show that the solution significantly improves the task success rate by up to
14%.

基于在线学习的用户中心化模型选择问题的解决方案通过结合开源编码器输出上下文和处理该上下文的在线学习算法，改善了选择开源和闭源模型之间的权衡，从而大幅提高了任务成功率。