Rare categories abound in a number of real-world networks and play a pivotal
role in a variety of high-stakes applications, including financial fraud
detection, network intrusion detection, and rare disease diagnosis. Rare
category analysis (RCA) refers to the task of detecting, characterizing, and
comprehending the behaviors of minority classes in a highly-imbalanced data
distribution. While the vast majority of existing work on RCA has focused on
improving the prediction performance, a few fundamental research questions
heretofore have received little attention and are less explored: How confident
or uncertain is a prediction model in rare category analysis? How can we
quantify the uncertainty in the learning process and enable reliable rare
category analysis?
To answer these questions, we start by investigating miscalibration in
existing RCA methods. Empirical results reveal that state-of-the-art RCA
methods are mainly over-confident in predicting minority classes and
under-confident in predicting majority classes. Motivated by the observation,
we propose a novel individual calibration framework, named CALIRARE, for
alleviating the unique challenges of RCA, thus enabling reliable rare category
analysis. In particular, to quantify the uncertainties in RCA, we develop a
node-level uncertainty quantification algorithm to model the overlapping
support regions with high uncertainty; to handle the rarity of minority classes
in miscalibration calculation, we generalize the distribution-based calibration
metric to the instance level and propose the first individual calibration
measurement on graphs named Expected Individual Calibration Error (EICE). We
perform extensive experimental evaluations on real-world datasets, including
rare category characterization and model calibration tasks, which demonstrate
the significance of our proposed framework.

在高度不平衡的数据分布中，研究了罕见类别分析（Rare Category Analysis）的不确定性量化、误校正和个体校准等问题，并提出了一种名为 CALIRARE 的新型个体校准框架来实现可靠的罕见类别分析。

通过个体校准实现对图上稀有类别的可靠分析

Towards Reliable Rare Category Analysis on Graphs via Individual  Calibration

In this paper, we consider the uncertainty quantification problem for
regression models. Specifically, we consider an individual calibration
objective for characterizing the quantiles of the prediction model. While such
an objective is well-motivated from downstream tasks such as newsvendor cost,
the existing methods have been largely heuristic and lack of statistical
guarantee in terms of individual calibration. We show via simple examples that
the existing methods focusing on population-level calibration guarantees such
as average calibration or sharpness can lead to harmful and unexpected results.
We propose simple nonparametric calibration methods that are agnostic of the
underlying prediction model and enjoy both computational efficiency and
statistical consistency. Our approach enables a better understanding of the
possibility of individual calibration, and we establish matching upper and
lower bounds for the calibration error of our proposed methods. Technically,
our analysis combines the nonparametric analysis with a covering number
argument for parametric analysis, which advances the existing theoretical
analyses in the literature of nonparametric density estimation and quantile
bandit problems. Importantly, the nonparametric perspective sheds new
theoretical insights into regression calibration in terms of the curse of
dimensionality and reconciles the existing results on the impossibility of
individual calibration. Numerical experiments show the advantage of such a
simple approach under various metrics, and also under covariates shift. We hope
our work provides a simple benchmark and a starting point of theoretical ground
for future research on regression calibration.

本文提出了针对回归模型的不确定性量化问题的非参数校准方法，该方法能够简单高效地提升模型个体校准的性能。通过数值实验，证明其优越性。

非参数方法下的无分布模型无关回归校准

Distribution-Free Model-Agnostic Regression Calibration via  Nonparametric Methods

Most machine learning classifiers only concern classification accuracy, while
certain applications (such as medical diagnosis, meteorological forecasting,
and computation advertising) require the model to predict the true probability,
known as a calibrated estimate. In previous work, researchers have developed
several calibration methods to post-process the outputs of a predictor to
obtain calibrated values, such as binning and scaling methods. Compared with
scaling, binning methods are shown to have distribution-free theoretical
guarantees, which motivates us to prefer binning methods for calibration.
However, we notice that existing binning methods have several drawbacks: (a)
the binning scheme only considers the original prediction values, thus limiting
the calibration performance; and (b) the binning approach is non-individual,
mapping multiple samples in a bin to the same value, and thus is not suitable
for order-sensitive applications. In this paper, we propose a feature-aware
binning framework, called Multiple Boosting Calibration Trees (MBCT), along
with a multi-view calibration loss to tackle the above issues. Our MBCT
optimizes the binning scheme by the tree structures of features, and adopts a
linear function in a tree node to achieve individual calibration. Our MBCT is
non-monotonic, and has the potential to improve order accuracy, due to its
learnable binning scheme and the individual calibration. We conduct
comprehensive experiments on three datasets in different fields. Results show
that our method outperforms all competing models in terms of both calibration
error and order accuracy. We also conduct simulation experiments, justifying
that the proposed multi-view calibration loss is a better metric in modeling
calibration error.

本文提出了一种称为多项提升校准树（MBCT）的特征感知分箱框架，针对机器学习分类器中缺乏对校准精度的处理，该校准过程通过学习多维特征，实现了非单调且可提升预测准确性的分箱校准方法。

MBCT: 面向特征的基于树的分箱方法用于个体不确定性校准

MBCT: Tree-Based Feature-Aware Binning for Individual Uncertainty  Calibration

Machine learning applications often require calibrated predictions, e.g. a
90\% credible interval should contain the true outcome 90\% of the times.
However, typical definitions of calibration only require this to hold on
average, and offer no guarantees on predictions made on individual samples.
Thus, predictions can be systematically over or under confident on certain
subgroups, leading to issues of fairness and potential vulnerabilities. We show
that calibration for individual samples is possible in the regression setup if
the predictions are randomized, i.e. outputting randomized credible intervals.
Randomization removes systematic bias by trading off bias with variance. We
design a training objective to enforce individual calibration and use it to
train randomized regression functions. The resulting models are more calibrated
for arbitrarily chosen subgroups of the data, and can achieve higher utility in
decision making against adversaries that exploit miscalibrated predictions.

通过使用随机化的置信区间实现对每个样本测量的分配，并提高数据子组的校准度和预测效果。