In this paper, we present a novel sensitivity-based filter pruning algorithm
(SbF-Pruner) to learn the importance scores of filters of each layer
end-to-end. Our method learns the scores from the filter weights, enabling it
to account for the correlations between the filters of each layer. Moreover, by
training the pruning scores of all layers simultaneously our method can account
for layer interdependencies, which is essential to find a performant sparse
sub-network. Our proposed method can train and generate a pruned network from
scratch in a straightforward, one-stage training process without requiring a
pretrained network. Ultimately, we do not need layer-specific hyperparameters
and pre-defined layer budgets, since SbF-Pruner can implicitly determine the
appropriate number of channels in each layer. Our experimental results on
different network architectures suggest that SbF-Pruner outperforms advanced
pruning methods. Notably, on CIFAR-10, without requiring a pretrained baseline
network, we obtain 1.02% and 1.19% accuracy gain on ResNet56 and ResNet110,
compared to the baseline reported for state-of-the-art pruning algorithms. This
is while SbF-Pruner reduces parameter-count by 52.3% (for ResNet56) and 54%
(for ResNet101), which is better than the state-of-the-art pruning algorithms
with a high margin of 9.5% and 6.6%.

本文提出了一种新颖的敏感度过滤剪枝算法（SbF-Pruner），可以通过端到端的方式学习每个层的过滤器重要性分数，从而生成性能更优异的稀疏子网络，并且完全不需要预先定义的超参数和层预算。

端到端基于敏感度的滤波剪枝

End-to-End Sensitivity-Based Filter Pruning

Domain classification is the fundamental task in natural language
understanding (NLU), which often requires fast accommodation to new emerging
domains. This constraint makes it impossible to retrain all previous domains,
even if they are accessible to the new model. Most existing continual learning
approaches suffer from low accuracy and performance fluctuation, especially
when the distributions of old and new data are significantly different. In
fact, the key real-world problem is not the absence of old data, but the
inefficiency to retrain the model with the whole old dataset. Is it potential
to utilize some old data to yield high accuracy and maintain stable
performance, while at the same time, without introducing extra hyperparameters?
In this paper, we proposed a hyperparameter-free continual learning model for
text data that can stably produce high performance under various environments.
Specifically, we utilize Fisher information to select exemplars that can
"record" key information of the original model. Also, a novel scheme called
dynamical weight consolidation is proposed to enable hyperparameter-free
learning during the retrain process. Extensive experiments demonstrate that
baselines suffer from fluctuated performance and therefore useless in practice.
On the contrary, our proposed model CCFI significantly and consistently
outperforms the best state-of-the-art method by up to 20% in average accuracy,
and each component of CCFI contributes effectively to overall performance.

本文提出了一个基于 Fisher 信息和动态权重合并的超参数自由连续学习模型来解决领域分类中的问题，并在实验中证明其在各种情况下可以稳定地产生高性能。

自然语言理解中无需超参数的领域分类连续学习

Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding

In Test-time Adaptation (TTA), given a source model, the goal is to adapt it
to make better predictions for test instances from a different distribution
than the source. Crucially, TTA assumes no access to the source data or even
any additional labeled/unlabeled samples from the target distribution to
finetune the source model. In this work, we consider TTA in a more pragmatic
setting which we refer to as SITA (Single Image Test-time Adaptation). Here,
when making a prediction, the model has access only to the given single test
instance, rather than a batch of instances, as typically been considered in the
literature. This is motivated by the realistic scenarios where inference is
needed on-demand instead of delaying for an incoming batch or the inference is
happening on an edge device (like mobile phone) where there is no scope for
batching. The entire adaptation process in SITA should be extremely fast as it
happens at inference time. To address this, we propose a novel approach AugBN
that requires only a single forward pass. It can be used on any off-the-shelf
trained model to test single instances for both classification and segmentation
tasks. AugBN estimates normalization statistics of the unseen test distribution
from the given test image using only one forward pass with label-preserving
transformations. Since AugBN does not involve any back-propagation, it is
significantly faster compared to recent test time adaptation methods. We
further extend AugBN to make the algorithm hyperparameter-free. Rigorous
experimentation show that our simple algorithm is able to achieve significant
performance gains for a variety of datasets, tasks, and network architectures.

该论文主要研究了用于单镜头图像推理的快速测试时间自适应方法（SITA）。 提出了一种名为 AugBN 的方法，它只需要一次前向传递来估计未见过的测试分布的规范化统计，该方法无需反向传播并且是无超参数的。 在实验中，该算法在各种数据集、任务和网络架构上都取得了显著的性能提升。

SITA: 单图测试时适应

SITA: Single Image Test-time Adaptation

We propose an online convex optimization algorithm (RescaledExp) that
achieves optimal regret in the unconstrained setting without prior knowledge of
any bounds on the loss functions. We prove a lower bound showing an exponential
separation between the regret of existing algorithms that require a known bound
on the loss functions and any algorithm that does not require such knowledge.
RescaledExp matches this lower bound asymptotically in the number of
iterations. RescaledExp is naturally hyperparameter-free and we demonstrate
empirically that it matches prior optimization algorithms that require
hyperparameter optimization.

我们提出了一种在线凸优化算法 (RescaledExp)，该算法在无约束设置下实现了最优的后悔，并没有先验知识任何损失函数的界限。我们证明了一个下界，说明现有算法在损失函数需要已知界限的情况下与不需要这种知识的任何算法之间存在指数分离。RescaledExp 在迭代次数方面与这个下界是一致的。RescaledExp 自然无超参数，并且我们通过实验证明它可以匹配需要超参数优化的之前的优化算法。