Training Deep Neural Networks (DNNs) is resource-intensive and
time-consuming. While prior research has explored many different ways of
reducing DNN training time, the impact of input data pipeline, i.e., fetching
raw data items from storage and performing data pre-processing in memory, has
been relatively unexplored. This paper makes the following contributions: (1)
We present the first comprehensive analysis of how the input data pipeline
affects the training time of widely-used computer vision and audio Deep Neural
Networks (DNNs), that typically involve complex data preprocessing. We analyze
nine different models across three tasks and four datasets while varying
factors such as the amount of memory, number of CPU threads, storage device,
GPU generation etc on servers that are a part of a large production cluster at
Microsoft. We find that in many cases, DNN training time is dominated by data
stall time: time spent waiting for data to be fetched and preprocessed. (2) We
build a tool, DS-Analyzer to precisely measure data stalls using a differential
technique, and perform predictive what-if analysis on data stalls. (3) Finally,
based on the insights from our analysis, we design and implement three simple
but effective techniques in a data-loading library, CoorDL, to mitigate data
stalls. Our experiments on a range of DNN tasks, models, datasets, and hardware
configs show that when PyTorch uses CoorDL instead of the state-of-the-art DALI
data loading library, DNN training time is reduced significantly (by as much as
5x on a single server).

本文针对输入数据管道对训练 DNN 模型的影响展开研究，使用 DS-Analyzer 工具精确测量数据阻塞时间，提出 CoorDL 数据加载库来缓解数据阻塞问题，并在多个 DNN 模型、数据集、硬件配置上进行验证，结果显示相对于 DALI 数据加载库，使用 CoorDL 数据加载库可以显著降低 DNN 训练时间。