We formulate the problem of constructing multiple simultaneously valid
confidence intervals (CIs) as estimating a high probability upper bound on the
maximum error for a class/set of estimate-estimand-error tuples, and refer to
this as the error estimation problem. For a single such tuple, data-driven
confidence intervals can often be used to bound the error in our estimate.
However, for a class of estimate-estimand-error tuples, nontrivial high
probability upper bounds on the maximum error often require class complexity as
input -- limiting the practicality of such methods and often resulting in loose
bounds. Rather than deriving theoretical class complexity-based bounds, we
propose a completely data-driven approach to estimate an upper bound on the
maximum error. The simple and general nature of our solution to this
fundamental challenge lends itself to several applications including: multiple
CI construction, multiple hypothesis testing, estimating excess risk bounds (a
fundamental measure of uncertainty in machine learning) for any
training/fine-tuning algorithm, and enabling the development of a contextual
bandit pipeline that can leverage any reward model estimation procedure as
input (without additional mathematical analysis).

通过数据驱动的方法，我们提出了一个完全基于数据的途径来估计最大误差的上界，解决了构建多个同时有效置信区间的问题，并拓展了估计过程的应用领域。