Recent breakthroughs in deep neural networks (DNNs) have fueled a tremendous
demand for intelligent edge devices featuring on-site learning, while the
practical realization of such systems remains a challenge due to the limited
resources available at the edge and the required massive training costs for
state-of-the-art (SOTA) DNNs. As reducing precision is one of the most
effective knobs for boosting training time/energy efficiency, there has been a
growing interest in low-precision DNN training. In this paper, we explore from
an orthogonal direction: how to fractionally squeeze out more training cost
savings from the most redundant bit level, progressively along the training
trajectory and dynamically per input. Specifically, we propose FracTrain that
integrates (i) progressive fractional quantization which gradually increases
the precision of activations, weights, and gradients that will not reach the
precision of SOTA static quantized DNN training until the final training stage,
and (ii) dynamic fractional quantization which assigns precisions to both the
activations and gradients of each layer in an input-adaptive manner, for only
"fractionally" updating layer parameters. Extensive simulations and ablation
studies (six models, four datasets, and three training settings including
standard, adaptation, and fine-tuning) validate the effectiveness of FracTrain
in reducing computational cost and hardware-quantified energy/latency of DNN
training while achieving a comparable or better (-0.12%~+1.87%) accuracy. For
example, when training ResNet-74 on CIFAR-10, FracTrain achieves 77.6% and
53.5% computational cost and training latency savings, respectively, compared
with the best SOTA baseline, while achieving a comparable (-0.07%) accuracy.
Our codes are available at: this https URL

本研究提出了一种名为 FracTrain 的系统，通过渐进式的分数量化和动态分数量化的方式来降低神经网络训练的计算成本和硬件开销，同时实现可比拟或更高的准确率。

FracTrain: 为高效 DNN 训练在时间和空间上实现分数压缩的比特节省

FracTrain: Fractionally Squeezing Bit Savings Both Temporally and  Spatially for Efficient DNN Training

We present a foveated object detector (FOD) as a biologically-inspired
alternative to the sliding window (SW) approach which is the dominant method of
search in computer vision object detection. Similar to the human visual system,
the FOD has higher resolution at the fovea and lower resolution at the visual
periphery. Consequently, more computational resources are allocated at the
fovea and relatively fewer at the periphery. The FOD processes the entire
scene, uses retino-specific object detection classifiers to guide eye
movements, aligns its fovea with regions of interest in the input image and
integrates observations across multiple fixations. Our approach combines modern
object detectors from computer vision with a recent model of peripheral pooling
regions found at the V1 layer of the human visual system. We assessed various
eye movement strategies on the PASCAL VOC 2007 dataset and show that the FOD
performs on par with the SW detector while bringing significant computational
cost savings.

本文提出了一种生物启发的凹视检测器（FOD）作为在计算机视觉目标检测中搜索的主要方法中滑动窗口 (SW) 方法的替代方案。这种方法通过结合现代计算机视觉中的目标检测器和人类视觉系统的 V1 层中发现的外围汇集区域的最近模型，实现了显着的计算成本节省。