A preference-based subjective evaluation is a key method for evaluating
generative media reliably. However, its huge combinations of pairs prohibit it
from being applied to large-scale evaluation using crowdsourcing. To address
this issue, we propose an automatic optimization method for preference-based
subjective evaluation in terms of pair combination selections and allocation of
evaluation volumes with online learning in a crowdsourcing environment. We use
a preference-based online learning method based on a sorting algorithm to
identify the total order of evaluation targets with minimum sample volumes. Our
online learning algorithm supports parallel and asynchronous execution under
fixed-budget conditions required for crowdsourcing. Our experiment on
preference-based subjective evaluation of synthetic speech shows that our
method successfully optimizes the test by reducing pair combinations from 351
to 83 and allocating optimal evaluation volumes for each pair ranging from 30
to 663 without compromising evaluation accuracies and wasting budget
allocations.

通过在线学习算法，我们提出了一种自动优化的方法来解决基于偏好的主观评估在众包环境中大规模测试时的配对组合选择和评估数量的分配问题。实验结果表明，我们的方法成功地通过减少配对组合数量并为每个配对分配最佳的评估数量来优化测试，同时不损害评估准确性和预算分配的浪费。

基于在线学习的众包环境下基于偏好的主观评估的自动设计优化

Automatic design optimization of preference-based subjective evaluation  with online learning in crowdsourcing environment

Transferable adversarial examples raise critical security concerns in
real-world, black-box attack scenarios. However, in this work, we identify two
main problems in common evaluation practices: (1) For attack transferability,
lack of systematic, one-to-one attack comparison and fair hyperparameter
settings. (2) For attack stealthiness, simply no comparisons. To address these
problems, we establish new evaluation guidelines by (1) proposing a novel
attack categorization strategy and conducting systematic and fair
intra-category analyses on transferability, and (2) considering diverse
imperceptibility metrics and finer-grained stealthiness characteristics from
the perspective of attack traceback. To this end, we provide the first
large-scale evaluation of transferable adversarial examples on ImageNet,
involving 23 representative attacks against 9 representative defenses. Our
evaluation leads to a number of new insights, including consensus-challenging
ones: (1) Under a fair attack hyperparameter setting, one early attack method,
DI, actually outperforms all the follow-up methods. (2) A state-of-the-art
defense, DiffPure, actually gives a false sense of (white-box) security since
it is indeed largely bypassed by our (black-box) transferable attacks. (3) Even
when all attacks are bounded by the same $L_p$ norm, they lead to dramatically
different stealthiness performance, which negatively correlates with their
transferability performance. Overall, our work demonstrates that existing
problematic evaluations have indeed caused misleading conclusions and missing
points, and as a result, hindered the assessment of the actual progress in this
field.

通过建立新的评估准则，我们在 ImageNet 上对 23 种典型攻击与 9 种代表性防御进行了首次大规模的可传递对抗样本评估，发现既有的评估存在误导性结论和遗漏点，从而阻碍了该领域的实际进展评估。