The travel time data collected from widespread traffic monitoring sensors
necessitate big data analytic tools for querying, visualization, and
identifying meaningful traffic patterns. This paper utilizes a large-scale
travel time dataset from Caltrans Performance Measurement System (PeMS) system
that is an overflow for traditional data processing and modeling tools. To
overcome the challenges of the massive amount of data, the big data analytic
engines Apache Spark and Apache MXNet are applied for data wrangling and
modeling. Seasonality and autocorrelation were performed to explore and
visualize the trend of time-varying data. Inspired by the success of the
hierarchical architecture for many Artificial Intelligent (AI) tasks, we
consolidate the cell and hidden states passed from low-level to the high-level
LSTM with an attention pooling similar to how the human perception system
operates. The designed hierarchical LSTM model can consider the dependencies at
different time scales to capture the spatial-temporal correlations of
network-level travel time. Another self-attention module is then devised to
connect LSTM extracted features to the fully connected layers, predicting
travel time for all corridors instead of a single link/route. The comparison
results show that the Hierarchical LSTM with Attention (HierLSTMat) model gives
the best prediction results at 30-minute and 45-min horizons and can
successfully forecast unusual congestion. The efficiency gained from big data
analytic tools was evaluated by comparing them with popular data science and
deep learning frameworks.

本文通过使用 Apache Spark 和 Apache MXNet 等大数据分析引擎，利用 Caltrans PeMS 系统的大规模旅行时间数据集设计了一个 Hierarchical LSTM with Attention (HierLSTMat) 模型来预测整个路网的旅行时间。实验结果表明该模型可以在多个计算时间内成功预测未知拥堵情况，并获得了比流行数据科学和深度学习框架更高的效率。