Prompting language models to provide step-by-step answers (e.g.,
"Chain-of-Thought") is the prominent approach for complex reasoning tasks,
where more accurate reasoning chains typically improve downstream task
performance. Recent literature discusses automatic methods to verify reasoning
steps to evaluate and improve their correctness. However, no fine-grained
step-level datasets are available to enable thorough evaluation of such
verification methods, hindering progress in this direction. We introduce
Reveal: Reasoning Verification Evaluation, a new dataset to benchmark automatic
verifiers of complex Chain-of-Thought reasoning in open-domain question
answering settings. Reveal includes comprehensive labels for the relevance,
attribution to evidence passages, and logical correctness of each reasoning
step in a language model's answer, across a wide variety of datasets and
state-of-the-art language models.

我们引入了 Reveal：推理验证评估，这是一个新的数据集，用于在开放域问题回答设置中评估复杂的链式推理的自动验证器，包括语言模型的答案中每个推理步骤的相关性、证据归因和逻辑正确性的全面标签。

思维链的强弱取决于最薄弱一环：推理链验证器的基准评估

A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for  Verifiers of Reasoning Chains

Safety measures need to be systemically investigated to what extent they
evaluate the intended performance of Deep Neural Networks (DNNs) for critical
applications. Due to a lack of verification methods for high-dimensional DNNs,
a trade-off is needed between accepted performance and handling of
out-of-distribution (OOD) samples.
This work evaluates rejecting outputs from semantic segmentation DNNs by
applying a Mahalanobis distance (MD) based on the most probable
class-conditional Gaussian distribution for the predicted class as an OOD
score. The evaluation follows three DNNs trained on the Cityscapes dataset and
tested on four automotive datasets and finds that classification risk can
drastically be reduced at the cost of pixel coverage, even when applied on
unseen datasets. The applicability of our findings will support legitimizing
safety measures and motivate their usage when arguing for safe usage of DNNs in
automotive perception.

通过应用基于最可能的类条件高斯分布的 Mahalanobis 距离作为 OOD 分数来评估拒绝语义分割 DNN 的输出，该研究发现即使在未知数据集上应用，也可以显著降低分类风险，但会以像素覆盖率为代价。这些发现的适用性将有助于合法化安全措施并推动其在汽车感知中的安全使用的论证。

自动驾驶数据集上的离群样本检测性能评估

Evaluation of Out-of-Distribution Detection Performance on Autonomous  Driving Datasets

A wide range of verification methods have been proposed to verify the safety
properties of deep neural networks ensuring that the networks function
correctly in critical applications. However, many well-known verification tools
still struggle with complicated network architectures and large network sizes.
In this work, we propose a network reduction technique as a pre-processing
method prior to verification. The proposed method reduces neural networks via
eliminating stable ReLU neurons, and transforming them into a sequential neural
network consisting of ReLU and Affine layers which can be handled by the most
verification tools. We instantiate the reduction technique on the
state-of-the-art complete and incomplete verification tools, including
alpha-beta-crown, VeriNet and PRIMA. Our experiments on a large set of
benchmarks indicate that the proposed technique can significantly reduce neural
networks and speed up existing verification tools. Furthermore, the experiment
results also show that network reduction can improve the availability of
existing verification tools on many networks by reducing them into sequential
neural networks.

通过消除稳定的 ReLU 神经元并将其转化为由 ReLU 和仿射层组成的顺序神经网络，我们提出了一种网络削减技术作为预处理方法以便于验证，实验证明这种技术能够显著减小神经网络规模并加快验证过程。

通过网络简化加快神经网络验证

Expediting Neural Network Verification via Network Reduction

To use neural networks in safety-critical settings it is paramount to provide
assurances on their runtime operation. Recent work on ReLU networks has sought
to verify whether inputs belonging to a bounded box can ever yield some
undesirable output. Input-splitting procedures, a particular type of
verification mechanism, do so by recursively partitioning the input set into
smaller sets. The efficiency of these methods is largely determined by the
number of splits the box must undergo before the property can be verified. In
this work, we propose a new technique based on shadow prices that fully
exploits the information of the problem yielding a more efficient generation of
splits than the state-of-the-art. Results on the Airborne Collision Avoidance
System (ACAS) benchmark verification tasks show a considerable reduction in the
partitions generated which substantially reduces computation times. These
results open the door to improved verification methods for a wide variety of
machine learning applications including vision and control.

本研究介绍了一种基于影子价格的新技术，可以更高效地对神经网络进行验证，降低了计算时间，同时也为视觉和控制等各种机器学习应用提供了改进的验证方法。