In an era where symbolic mathematical equations are indispensable for
modeling complex natural phenomena, scientific inquiry often involves
collecting observations and translating them into mathematical expressions.
Recently, deep learning has emerged as a powerful tool for extracting insights
from data. However, existing models typically specialize in either numeric or
symbolic domains, and are usually trained in a supervised manner tailored to
specific tasks. This approach neglects the substantial benefits that could
arise from a task-agnostic unified understanding between symbolic equations and
their numeric counterparts. To bridge the gap, we introduce SNIP, a
Symbolic-Numeric Integrated Pre-training, which employs joint contrastive
learning between symbolic and numeric domains, enhancing their mutual
similarities in the pre-trained embeddings. By performing latent space
analysis, we observe that SNIP provides cross-domain insights into the
representations, revealing that symbolic supervision enhances the embeddings of
numeric data and vice versa. We evaluate SNIP across diverse tasks, including
symbolic-to-numeric mathematical property prediction and numeric-to-symbolic
equation discovery, commonly known as symbolic regression. Results show that
SNIP effectively transfers to various tasks, consistently outperforming fully
supervised baselines and competing strongly with established task-specific
methods, especially in few-shot learning scenarios where available data is
limited.

在一个符号化数学方程与其数值对应物之间具有任务不可知的统一理解的显著益处的基础上，我们介绍了 SNIP，一种符号 - 数字集成预训练方法，通过使用符号领域和数字领域之间的联合对比学习来增强它们在预训练嵌入中的相似性。通过进行潜空间分析，我们观察到 SNIP 提供了对表示的跨领域见解，揭示了符号监督如何增强数字数据的嵌入以及反之。我们在包括符号到数值数学性质预测和数值到符号方程发现 (通常被称为符号回归) 在内的各种任务上评估了 SNIP。结果表明，SNIP 能够有效地应用于各种任务，在有限数据的少样本学习场景中始终优于全监督基线并与已确立的任务特定方法竞争激烈。

SNIP: 数学符号与数值领域的统一预训练桥梁

SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified  Pre-training

Recent work has explored the possibility of pruning neural networks at
initialization. We assess proposals for doing so: SNIP (Lee et al., 2019),
GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude
pruning. Although these methods surpass the trivial baseline of random pruning,
they remain below the accuracy of magnitude pruning after training, and we
endeavor to understand why. We show that, unlike pruning after training,
randomly shuffling the weights these methods prune within each layer or
sampling new initial values preserves or improves accuracy. As such, the
per-weight pruning decisions made by these methods can be replaced by a
per-layer choice of the fraction of weights to prune. This property suggests
broader challenges with the underlying pruning heuristics, the desire to prune
at initialization, or both.

研究表明神经网络在初始化阶段进行剪枝是可能的，但是现有方法（SNIP、GraSP、SynFlow 和 magnitude pruning）表现不如训练后的 magnitude pruning，可能是因为这些方法的权重剪枝决策可以通过每层选择剪枝权重比例来替换，这反映出底层剪枝启发式算法及初始化阶段进行剪枝的挑战。

神经网络初始化时的剪枝：为什么我们会错过目标？

Pruning Neural Networks at Initialization: Why are We Missing the Mark?

With the introduction of SNIP [arXiv:1810.02340v2], it has been demonstrated
that modern neural networks can effectively be pruned before training. Yet, its
sensitivity criterion has since been criticized for not propagating training
signal properly or even disconnecting layers. As a remedy, GraSP
[arXiv:2002.07376v1] was introduced, compromising on simplicity. However, in
this work we show that by applying the sensitivity criterion iteratively in
smaller steps - still before training - we can improve its performance without
difficult implementation. As such, we introduce 'SNIP-it'. We then demonstrate
how it can be applied for both structured and unstructured pruning, before
and/or during training, therewith achieving state-of-the-art
sparsity-performance trade-offs. That is, while already providing the
computational benefits of pruning in the training process from the start.
Furthermore, we evaluate our methods on robustness to overfitting,
disconnection and adversarial attacks as well.

本文介绍了 SNIP-it，一种在训练前通过迭代敏感性准则进行剪枝的方法，提高了现代神经网络的剪枝表现，同时具有抗过拟合、抗断开和抗对抗攻击等特点。

通过敏感度统计量的迭代排名进行剪枝

Pruning via Iterative Ranking of Sensitivity Statistics

An analysis of different techniques for recognizing and detecting objects
under extreme scale variation is presented. Scale specific and scale invariant
design of detectors are compared by training them with different configurations
of input data. By evaluating the performance of different network architectures
for classifying small objects on ImageNet, we show that CNNs are not robust to
changes in scale. Based on this analysis, we propose to train and test
detectors on the same scales of an image-pyramid. Since small and large objects
are difficult to recognize at smaller and larger scales respectively, we
present a novel training scheme called Scale Normalization for Image Pyramids
(SNIP) which selectively back-propagates the gradients of object instances of
different sizes as a function of the image scale. On the COCO dataset, our
single model performance is 45.7% and an ensemble of 3 networks obtains an mAP
of 48.3%. We use off-the-shelf ImageNet-1000 pre-trained models and only train
with bounding box supervision. Our submission won the Best Student Entry in the
COCO 2017 challenge. Code will be made available at
http://bit.ly/2yXVg4c.

本文分析了不同的目标识别和检测技术，针对极端尺度变化进行了比较，比较了尺度特定和尺度不变的设计，提出了一种新的训练方案：图像金字塔尺度归一化检测器（SNIP），并在 COCO 数据集上进行了评估。