Accelerating Human Action Recognition (HAR) efficiently for real-time
surveillance and robotic systems on edge chips remains a challenging research
field, given its high computational and memory requirements. This paper
proposed an integrated end-to-end HAR scalable HW/SW accelerator co-design
based on an enhanced 8-bit quantized Two-Stream SimpleNet-PyTorch CNN
architecture. Our network accelerator was trained on UCF101 and UCF24 datasets
and implemented on edge SoC-FPGA. Our development uses partially streaming
dataflow architecture to achieve higher throughput versus network design and
resource utilization trade-off. We also fused all convolutional, batch-norm,
and ReLU operations into a single homogeneous layer and utilized the
Lucas-Kanade motion flow method to enable a high parallelism accelerator design
and optimized on-chip engine computing.Furthermore, our proposed methodology
achieved nearly 81% prediction accuracy with an approximately 24 FPS real-time
inference throughput at 187MHz on ZCU104, which is 1.7x - 1.9x higher than the
prior research. Lastly, the designed framework was benchmarked against several
hardware chips for higher throughput and performance measurements and is now
available as an open-source project on GitHub for training and implementation
on edge platforms.

提出了一种基于改进的 8 位量化的 Two-Stream SimpleNet-PyTorch CNN 体系结构的集成端到端的可扩展硬件 / 软件加速器共设计用于加速实时监控和机器人系统上的人类动作识别。通过将所有卷积、批量归一化和 ReLU 操作融合成一个单一的均匀层，并利用 Lucas-Kanade 运动流方法实现高并行加速器设计和优化的芯片上引擎计算，实现了近 81% 的预测准确率和约 24 FPS 的实时推理吞吐量，其性能比以前的研究提高了 1.7 倍到 1.9 倍。该框架针对更高的吞吐量和性能测量进行了基准测试，并作为一个开源项目在 GitHub 上提供，用于训练和实现在边缘平台上。