Consider a scenario where we are supplied with a number of ready-to-use
models trained on a certain source domain and hope to directly apply the most
appropriate ones to different target domains based on the models' relative
performance. Ideally we should annotate a validation set for model performance
assessment on each new target environment, but such annotations are often very
expensive. Under this circumstance, we introduce the problem of ranking models
in unlabeled new environments. For this problem, we propose to adopt a proxy
dataset that 1) is fully labeled and 2) well reflects the true model rankings
in a given target environment, and use the performance rankings on the proxy
sets as surrogates. We first select labeled datasets as the proxy.
Specifically, datasets that are more similar to the unlabeled target domain are
found to better preserve the relative performance rankings. Motivated by this,
we further propose to search the proxy set by sampling images from various
datasets that have similar distributions as the target. We analyze the problem
and its solutions on the person re-identification (re-ID) task, for which
sufficient datasets are publicly available, and show that a carefully
constructed proxy set effectively captures relative performance ranking in new
environments. Code is available at https://github.com/sxzrt/Proxy-Set.

本研究提出了一种使用代理数据集来解决领域自适应问题的方法，该数据集经过良好构建可以有效捕捉到具有相似分布的目标数据集的模型性能排名，并在 Person Re-identification 任务中进行了验证。

无标签新环境中的排名模型

Ranking Models in Unlabeled New Environments

At the present time, sequential item recommendation models are compared by
calculating metrics on a small item subset (target set) to speed up
computation. The target set contains the relevant item and a set of negative
items that are sampled from the full item set. Two well-known strategies to
sample negative items are uniform random sampling and sampling by popularity to
better approximate the item frequency distribution in the dataset. Most
recently published papers on sequential item recommendation rely on sampling by
popularity to compare the evaluated models. However, recent work has already
shown that an evaluation with uniform random sampling may not be consistent
with the full ranking, that is, the model ranking obtained by evaluating a
metric using the full item set as target set, which raises the question whether
the ranking obtained by sampling by popularity is equal to the full ranking. In
this work, we re-evaluate current state-of-the-art sequential recommender
models from the point of view, whether these sampling strategies have an impact
on the final ranking of the models. We therefore train four recently proposed
sequential recommendation models on five widely known datasets. For each
dataset and model, we employ three evaluation strategies. First, we compute the
full model ranking. Then we evaluate all models on a target set sampled by the
two different sampling strategies, uniform random sampling and sampling by
popularity with the commonly used target set size of 100, compute the model
ranking for each strategy and compare them with each other. Additionally, we
vary the size of the sampled target set. Overall, we find that both sampling
strategies can produce inconsistent rankings compared with the full ranking of
the models. Furthermore, both sampling by popularity and uniform random
sampling do not consistently produce the same ranking ...

本研究针对目前流行的序列推荐模型的负样本采样方法进行重新评估，发现流行的负样本采样方法并不能保证与完整数据集的排名结果一致，建议结合多种评估策略进行排名比较。