Deep learning models have revolutionized various fields, from image
recognition to natural language processing, by achieving unprecedented levels
of accuracy. However, their increasing energy consumption has raised concerns
about their environmental impact, disadvantaging smaller entities in research
and exacerbating global energy consumption. In this paper, we explore the
trade-off between model accuracy and electricity consumption, proposing a
metric that penalizes large consumption of electricity. We conduct a
comprehensive study on the electricity consumption of various deep learning
models across different GPUs, presenting a detailed analysis of their
accuracy-efficiency trade-offs. By evaluating accuracy per unit of electricity
consumed, we demonstrate how smaller, more energy-efficient models can
significantly expedite research while mitigating environmental concerns. Our
results highlight the potential for a more sustainable approach to deep
learning, emphasizing the importance of optimizing models for efficiency. This
research also contributes to a more equitable research landscape, where smaller
entities can compete effectively with larger counterparts. This advocates for
the adoption of efficient deep learning practices to reduce electricity
consumption, safeguarding the environment for future generations whilst also
helping ensure a fairer competitive landscape.

我们研究了深度学习模型在准确性和电力消耗之间的权衡，并提出一种惩罚高电力消耗的度量标准。通过评估单位电力消耗下的准确性，我们展示了更小、更能源高效的模型如何在减缓环境问题的同时显著加快研究进展。这项研究强调了深度学习优化模型效率的重要性，为更公平的竞争环境做出了贡献。

Watt 为何物：重新思考深度学习的能耗和性能关系

Watt For What: Rethinking Deep Learning's Energy-Performance  Relationship

The width of a neural network matters since increasing the width will
necessarily increase the model capacity. However, the performance of a network
does not improve linearly with the width and soon gets saturated. In this case,
we argue that increasing the number of networks (ensemble) can achieve better
accuracy-efficiency trade-offs than purely increasing the width. To prove it,
one large network is divided into several small ones regarding its parameters
and regularization components. Each of these small networks has a fraction of
the original one's parameters. We then train these small networks together and
make them see various views of the same data to increase their diversity.
During this co-training process, networks can also learn from each other. As a
result, small networks can achieve better ensemble performance than the large
one with few or no extra parameters or FLOPs, \ie, achieving better
accuracy-efficiency trade-offs. Small networks can also achieve faster
inference speed than the large one by concurrent running. All of the above
shows that the number of networks is a new dimension of model scaling. We
validate our argument with 8 different neural architectures on common
benchmarks through extensive experiments. The code is available at
https://github.com/FreeformRobotics/Divide-and-Co-training.

通过将大型神经网络分解成多个小型网络，并通过共同训练和学习来提高网络的多样性，小型网络可以在不增加参数和 FLOPs 的情况下实现更好的集成性能和更快的推理速度。我们在八个不同的神经结构和常见基准测试中验证了该论点，代码可在 https://github.com/FreeformRobotics/Divide-and-Co-training 中获得。