In the last decade, Convolutional Neural Network with a multi-layer
architecture has advanced rapidly. However, training its complex network is
very space-consuming, since a lot of intermediate data are preserved across
layers, especially when processing high-dimension inputs with a big batch size.
That poses great challenges to the limited memory capacity of current
accelerators (e.g., GPUs). Existing efforts mitigate such bottleneck by
external auxiliary solutions with additional hardware costs, and internal
modifications with potential accuracy penalty. Differently, our analysis
reveals that computations intra- and inter-layers exhibit the spatial-temporal
weak dependency and even complete independency features. That inspires us to
break the traditional layer-by-layer (column) dataflow rule. Now operations are
novelly re-organized into rows throughout all convolution layers. This
lightweight design allows a majority of intermediate data to be removed without
any loss of accuracy. We particularly study the weak dependency between two
consecutive rows. For the resulting skewed memory consumption, we give two
solutions with different favorite scenarios. Evaluations on two representative
networks confirm the effectiveness. We also validate that our middle dataflow
optimization can be smoothly embraced by existing works for better memory
reduction.

我们通过重新组织操作，将传统的一层一层的数据流规则打破，设计了一种轻量级架构，能够在不影响准确性的情况下移除大部分中间数据，有效降低内存消耗。我们特别研究了两个连续行之间的较弱依赖关系，提出了两种解决方案，并通过评估验证了其有效性。我们还验证了我们的中间数据流优化可以顺利结合现有工作以获得更好的内存减少效果。