Data sampling is an effective method to improve the training speed of neural
networks, with recent results demonstrating that it can even break the neural
scaling laws. These results critically rely on high-quality scores to estimate
the importance of an input to the network. We observe that there are two
dominant strategies: static sampling, where the scores are determined before
training, and dynamic sampling, where the scores can depend on the model
weights. Static algorithms are computationally inexpensive but less effective
than their dynamic counterparts, which can cause end-to-end slowdown due to
their need to explicitly compute losses. To address this problem, we propose a
novel sampling distribution based on nonparametric kernel regression that
learns an effective importance score as the neural network trains. However,
nonparametric regression models are too computationally expensive to accelerate
end-to-end training. Therefore, we develop an efficient sketch-based
approximation to the Nadaraya-Watson estimator. Using recent techniques from
high-dimensional statistics and randomized algorithms, we prove that our
Nadaraya-Watson sketch approximates the estimator with exponential convergence
guarantees. Our sampling algorithm outperforms the baseline in terms of
wall-clock time and accuracy on four datasets.

通过使用非参数核回归进行采样，我们提出了一种新颖的采样分布，它能够在神经网络训练过程中学习到有效的重要性评分。我们的采样算法在墙钟时间和准确性上优于基准算法。

通过高效的非参数替代品实现深度学习的自适应采样

Adaptive Sampling for Deep Learning via Efficient Nonparametric Proxies

We propose universally slimmable self-supervised learning (dubbed as US3L) to
achieve better accuracy-efficiency trade-offs for deploying self-supervised
models across different devices. We observe that direct adaptation of
self-supervised learning (SSL) to universally slimmable networks misbehaves as
the training process frequently collapses. We then discover that temporal
consistent guidance is the key to the success of SSL for universally slimmable
networks, and we propose three guidelines for the loss design to ensure this
temporal consistency from a unified gradient perspective. Moreover, we propose
dynamic sampling and group regularization strategies to simultaneously improve
training efficiency and accuracy. Our US3L method has been empirically
validated on both convolutional neural networks and vision transformers. With
only once training and one copy of weights, our method outperforms various
state-of-the-art methods (individually trained or not) on benchmarks including
recognition, object detection and instance segmentation. Our code is available
at this https URL

我们提出了一种称为 US3L 的普遍可裁剪自监督学习方法，它通过确保从统一的梯度视角下的时间一致性来在不同设备上部署自监督模型时实现更好的准确性 - 效率权衡，并通过动态抽样和群组正则化策略同时提高训练效率和准确性。在卷积神经网络和视觉变换器上，我们的方法通过一次训练和一个权重副本就能胜过各种最先进的方法（单独训练或不训练），在识别、目标检测和实例分割等基准测试上进行了经验证明。

普适可调整自监督学习三项指南

Three Guidelines You Should Know for Universally Slimmable  Self-Supervised Learning

Federated learning (FL) is a novel machine learning setting that enables
on-device intelligence via decentralized training and federated optimization.
Deep neural networks' rapid development facilitates the learning techniques for
modeling complex problems and emerges into federated deep learning under the
federated setting. However, the tremendous amount of model parameters burdens
the communication network with a high load of transportation. This paper
introduces two approaches for improving communication efficiency by dynamic
sampling and top-$k$ selective masking. The former controls the fraction of
selected client models dynamically, while the latter selects parameters with
top-$k$ largest values of difference for federated updating. Experiments on
convolutional image classification and recurrent language modeling are
conducted on three public datasets to show our proposed methods' effectiveness.

本研究通过动态采样和 Top-K 选择性掩蔽这两种方法，提高了联邦学习的通信效率，并在公共数据集上进行实验，证明了该方法的有效性。

动态采样与选择性遮蔽：用于高效通信的联合学习

Dynamic Sampling and Selective Masking for Communication-Efficient  Federated Learning

Traditional Neural machine translation (NMT) involves a fixed training
procedure where each sentence is sampled once during each epoch. In reality,
some sentences are well-learned during the initial few epochs; however, using
this approach, the well-learned sentences would continue to be trained along
with those sentences that were not well learned for 10-30 epochs, which results
in a wastage of time. Here, we propose an efficient method to dynamically
sample the sentences in order to accelerate the NMT training. In this approach,
a weight is assigned to each sentence based on the measured difference between
the training costs of two iterations. Further, in each epoch, a certain
percentage of sentences are dynamically sampled according to their weights.
Empirical results based on the NIST Chinese-to-English and the WMT
English-to-German tasks depict that the proposed method can significantly
accelerate the NMT training and improve the NMT performance.

该研究提出了一种基于动态采样的神经机器翻译（NMT）训练方法，通过对句子加权并根据权重动态采样一定百分比的句子，以提高 NMT 训练效率和性能。