We derive an (almost) guaranteed upper bound on the error of deep neural
networks under distribution shift using unlabeled test data. Prior methods
either give bounds that are vacuous in practice or give estimates that are
accurate on average but heavily underestimate error for a sizeable fraction of
shifts. In particular, the latter only give guarantees based on complex
continuous measures such as test calibration -- which cannot be identified
without labels -- and are therefore unreliable. Instead, our bound requires a
simple, intuitive condition which is well justified by prior empirical works
and holds in practice effectively 100% of the time. The bound is inspired by
$\mathcal{H}\Delta\mathcal{H}$-divergence but is easier to evaluate and
substantially tighter, consistently providing non-vacuous guarantees.
Estimating the bound requires optimizing one multiclass classifier to disagree
with another, for which some prior works have used sub-optimal proxy losses; we
devise a "disagreement loss" which is theoretically justified and performs
better in practice. We expect this loss can serve as a drop-in replacement for
future methods which require maximizing multiclass disagreement. Across a wide
range of benchmarks, our method gives valid error bounds while achieving
average accuracy comparable to competitive estimation baselines. Code is
publicly available at this https URL .

本研究通过使用无标签测试数据来推导深度神经网络在分布转移下的误差（几乎）保证上界，并提出了一种理论上合理、在实践中表现更优的 “不一致损失” 用于评估误差保证上界的较紧估计。

(几乎) 通过分歧差异证明分布偏移下的误差界

(Almost) Provable Error Bounds Under Distribution Shift via Disagreement  Discrepancy

A multiclass classifier is said to be top-label calibrated if the reported
probability for the predicted class -- the top-label -- is calibrated,
conditioned on the top-label. This conditioning on the top-label is absent in
the closely related and popular notion of confidence calibration, which we
argue makes confidence calibration difficult to interpret for decision-making.
We propose top-label calibration as a rectification of confidence calibration.
Further, we outline a multiclass-to-binary (M2B) reduction framework that
unifies confidence, top-label, and class-wise calibration, among others. As its
name suggests, M2B works by reducing multiclass calibration to numerous binary
calibration problems, each of which can be solved using simple binary
calibration routines. We instantiate the M2B framework with the well-studied
histogram binning (HB) binary calibrator, and prove that the overall procedure
is multiclass calibrated without making any assumptions on the underlying data
distribution. In an empirical evaluation with four deep net architectures on
CIFAR-10 and CIFAR-100, we find that the M2B + HB procedure achieves lower
top-label and class-wise calibration error than other approaches such as
temperature scaling. Code for this work is available at
https://github.com/aigen/df-posthoc-calibration.

本文提出了一种名为顶级标签校准的新型多分类器校准方法，该方法通过将多分类校准转化为众多的二分类校准问题并使用简单二进制校准程序来解决，实验结果表明该方法在 CIFAR-10 和 CIFAR-100 上的表现比温度缩放等其他方法更加优秀。

高置信度标签校准和多类到二进制降维

Top-label calibration and multiclass-to-binary reductions

The growing use of IoT devices in organizations has increased the number of
attack vectors available to attackers due to the less secure nature of the
devices. The widely adopted bring your own device (BYOD) policy which allows an
employee to bring any IoT device into the workplace and attach it to an
organization's network also increases the risk of attacks. In order to address
this threat, organizations often implement security policies in which only the
connection of white-listed IoT devices is permitted. To monitor adherence to
such policies and protect their networks, organizations must be able to
identify the IoT devices connected to their networks and, more specifically, to
identify connected IoT devices that are not on the white-list (unknown
devices). In this study, we applied deep learning on network traffic to
automatically identify IoT devices connected to the network. In contrast to
previous work, our approach does not require that complex feature engineering
be applied on the network traffic, since we represent the communication
behavior of IoT devices using small images built from the IoT devices network
traffic payloads. In our experiments, we trained a multiclass classifier on a
publicly available dataset, successfully identifying 10 different IoT devices
and the traffic of smartphones and computers, with over 99% accuracy. We also
trained multiclass classifiers to detect unauthorized IoT devices connected to
the network, achieving over 99% overall average detection accuracy.

本研究采用深度学习方法分析网络流量，通过网络流量载荷生成小图像的方式，实现对连接到网络上的各种 IoT 设备及未授权连接的设备进行自动识别，训练出的多分类器在公开数据集上精度超过 99%。