Deep Learning (DL) has transformed the automation of a wide range of
industries and finds increasing ubiquity in society. The high complexity of DL
models and its widespread adoption has led to global energy consumption
doubling every 3-4 months. Currently, the relationship between the DL model
configuration and energy consumption is not well established. At a general
computational energy model level, there is both strong dependency to both the
hardware architecture (e.g. generic processors with different configuration of
inner components- CPU and GPU, programmable integrated circuits - FPGA), as
well as different interacting energy consumption aspects (e.g., data movement,
calculation, control). At the DL model level, we need to translate non-linear
activation functions and its interaction with data into calculation tasks.
Current methods mainly linearize nonlinear DL models to approximate its
theoretical FLOPs and MACs as a proxy for energy consumption. Yet, this is
inaccurate (est. 93\% accuracy) due to the highly nonlinear nature of many
convolutional neural networks (CNNs) for example.
In this paper, we develop a bottom-level Transistor Operations (TOs) method
to expose the role of non-linear activation functions and neural network
structure in energy consumption. We translate a range of feedforward and CNN
models into ALU calculation tasks and then TO steps. This is then statistically
linked to real energy consumption values via a regression model for different
hardware configurations and data sets. We show that our proposed TOs method can
achieve a 93.61% - 99.51% precision in predicting its energy consumption.

本文提出基于底层晶体管操作的方法来研究深度学习中非线性激活函数和神经网络结构对能量消耗的影响，实验证明该方法可以高度准确地预测深度学习模型的能量消耗。

用于深度学习能耗缩放律的晶体管操作模型

A Transistor Operations Model for Deep Learning Energy Consumption Scaling Law

Recently, deep learning-based algorithms are widely adopted due to the
advantage of being able to establish anomaly detection models without or with
minimal domain knowledge of the task. Instead, to train the artificial neural
network more stable, it should be better to define the appropriate neural
network structure or the loss function. For the training anomaly detection
model, the mean squared error (MSE) function is adopted widely. On the other
hand, the novel loss function, logarithmic mean squared error (LMSE), is
proposed in this paper to train the neural network more stable. This study
covers a variety of comparisons from mathematical comparisons, visualization in
the differential domain for backpropagation, loss convergence in the training
process, and anomaly detection performance. In an overall view, LMSE is
superior to the existing MSE function in terms of strongness of loss
convergence, anomaly detection performance. The LMSE function is expected to be
applicable for training not only the anomaly detection model but also the
general generative neural network.

提出了一种新的损失函数 LMSE，比现有的 MSE 函数更加稳定，对于异常检测性能和损失函数收敛性更强，在深度学习的领域中具有广泛的应用前景。