This paper explores Deep Learning (DL) methods that are used or have the
potential to be used for traffic video analysis, emphasizing driving safety for
both Autonomous Vehicles (AVs) and human-operated vehicles. We present a
typical processing pipeline, which can be used to understand and interpret
traffic videos by extracting operational safety metrics and providing general
hints and guidelines to improve traffic safety. This processing framework
includes several steps, including video enhancement, video stabilization,
semantic and incident segmentation, object detection and classification,
trajectory extraction, speed estimation, event analysis, modeling and anomaly
detection. Our main goal is to guide traffic analysts to develop their own
custom-built processing frameworks by selecting the best choices for each step
and offering new designs for the lacking modules by providing a comparative
analysis of the most successful conventional and DL-based algorithms proposed
for each step. We also review existing open-source tools and public datasets
that can help train DL models. To be more specific, we review exemplary traffic
problems and mentioned requires steps for each problem. Besides, we investigate
connections to the closely related research areas of drivers' cognition
evaluation, Crowd-sourcing-based monitoring systems, Edge Computing in roadside
infrastructures, Automated Driving Systems (ADS)-equipped vehicles, and
highlight the missing gaps. Finally, we review commercial implementations of
traffic monitoring systems, their future outlook, and open problems and
remaining challenges for widespread use of such systems.

本文探讨用于交通视频分析的深度学习方法，着重强调自动驾驶车辆和人类驾驶车辆的行驶安全。提出一个典型的处理流程，可以通过提取操作安全指标和提供通用提示和指南来理解和解释交通视频。主要包括视频增强、视频稳定、语义和事件分割、目标检测和分类、轨迹提取、速度估计、事件分析、建模和异常检测等几个步骤。重点在于选择最佳步骤并提供缺少模块的新设计来引导交通分析师开发他们自己的自定义处理框架。还回顾了现有的开源工具和公共数据集，以帮助训练深度学习模型。此外，还调查了与驾驶员认知评估、基于众包的监测系统、路边基础设施中的边缘计算、配备自动驾驶系统的汽车等相关的研究领域的联系和缺失方面。最后，回顾了交通监控系统的商业实施、未来展望以及广泛使用此类系统所面临的开放性问题和挑战。