Existing pedestrian behavior prediction methods rely primarily on deep neural
networks that utilize features extracted from video frame sequences. Although
these vision-based models have shown promising results, they face limitations
in effectively capturing and utilizing the dynamic spatio-temporal interactions
between the target pedestrian and its surrounding traffic elements, crucial for
accurate reasoning. Additionally, training these models requires manually
annotating domain-specific datasets, a process that is expensive,
time-consuming, and difficult to generalize to new environments and scenarios.
The recent emergence of Large Multimodal Models (LMMs) offers potential
solutions to these limitations due to their superior visual understanding and
causal reasoning capabilities, which can be harnessed through semi-supervised
training. GPT-4V(ision), the latest iteration of the state-of-the-art
Large-Language Model GPTs, now incorporates vision input capabilities. This
report provides a comprehensive evaluation of the potential of GPT-4V for
pedestrian behavior prediction in autonomous driving using publicly available
datasets: JAAD, PIE, and WiDEVIEW. Quantitative and qualitative evaluations
demonstrate GPT-4V(ision)'s promise in zero-shot pedestrian behavior prediction
and driving scene understanding ability for autonomous driving. However, it
still falls short of the state-of-the-art traditional domain-specific models.
Challenges include difficulties in handling small pedestrians and vehicles in
motion. These limitations highlight the need for further research and
development in this area.

GPT-4V 在自动驾驶中利用公开可用的数据集 JAAD、PIE 和 WiDEVIEW 进行行人行为预测的评估，定量和定性评估证明了 GPT-4V 在零 - shot 行人行为预测和自动驾驶中的驾驶场景理解能力，但仍不及传统领域特定模型的最新成果，存在小行人和运动车辆处理困难等限制，需要进一步研究和发展。

GPT-4V 掌控方向：对行人行为预测的承诺和挑战评估

GPT-4V Takes the Wheel: Evaluating Promise and Challenges for Pedestrian  Behavior Prediction

Predicting pedestrian behavior when interacting with vehicles is one of the
most critical challenges in the field of automated driving. Pedestrian crossing
behavior is influenced by various interaction factors, including time to
arrival, pedestrian waiting time, the presence of zebra crossing, and the
properties and personality traits of both pedestrians and drivers. However,
these factors have not been fully explored for use in predicting interaction
outcomes. In this paper, we use machine learning to predict pedestrian crossing
behavior including pedestrian crossing decision, crossing initiation time
(CIT), and crossing duration (CD) when interacting with vehicles at
unsignalized crossings. Distributed simulator data are utilized for predicting
and analyzing the interaction factors. Compared with the logistic regression
baseline model, our proposed neural network model improves the prediction
accuracy and F1 score by 4.46% and 3.23%, respectively. Our model also reduces
the root mean squared error (RMSE) for CIT and CD by 21.56% and 30.14% compared
with the linear regression model. Additionally, we have analyzed the importance
of interaction factors, and present the results of models using fewer factors.
This provides information for model selection in different scenarios with
limited input features.

本文利用机器学习预测在非信号控制的人行横道中行人与车辆的交互，包括行人的行走决策、开始过马路的时间和过马路所需时间。使用神经网络模型比 logistic 回归模型提高了 4.46% 的预测准确率和 3.23% 的 F1 分数，并与线性回归模型相比，降低了 21.56% 和 30.14% 的 CIT 和 CD 的根均方误差。分析了交互因素的重要性，提供了在不同特征输入限制下选择模型的信息。

无信号路口行人交互结果预测：等待还是穿过？

Cross or Wait? Predicting Pedestrian Interaction Outcomes at  Unsignalized Crossings

Pedestrian behavior prediction is one of the major challenges for intelligent
driving systems. Pedestrians often exhibit complex behaviors influenced by
various contextual elements. To address this problem, we propose BiPed, a
multitask learning framework that simultaneously predicts trajectories and
actions of pedestrians by relying on multimodal data. Our method benefits from
1) a bifold encoding approach where different data modalities are processed
independently allowing them to develop their own representations, and jointly
to produce a representation for all modalities using shared parameters; 2) a
novel interaction modeling technique that relies on categorical semantic
parsing of the scenes to capture interactions between target pedestrians and
their surroundings; and 3) a bifold prediction mechanism that uses both
independent and shared decoding of multimodal representations. Using public
pedestrian behavior benchmark datasets for driving, PIE and JAAD, we highlight
the benefits of the proposed method for behavior prediction and show that our
model achieves state-of-the-art performance and improves trajectory and action
prediction by up to 22% and 9% respectively. We further investigate the
contributions of the proposed reasoning techniques via extensive ablation
studies.

提出了一种多模态数据下的多任务学习框架 ——BiPed，其中包括一个双向编码器以及新颖的交互建模技术，同时提高了行人轨迹和行为预测的性能达到最新水平。

步行者行为预测的双向和语义推理

Bifold and Semantic Reasoning for Pedestrian Behavior Prediction

Pedestrian behavior prediction is one of the major challenges for intelligent
driving systems in urban environments. Pedestrians often exhibit a wide range
of behaviors and adequate interpretations of those depend on various sources of
information such as pedestrian appearance, states of other road users, the
environment layout, etc. To address this problem, we propose a novel
multi-modal prediction algorithm that incorporates different sources of
information captured from the environment to predict future crossing actions of
pedestrians. The proposed model benefits from a hybrid learning architecture
consisting of feedforward and recurrent networks for analyzing visual features
of the environment and dynamics of the scene. Using the existing 2D pedestrian
behavior benchmarks and a newly annotated 3D driving dataset, we show that our
proposed model achieves state-of-the-art performance in pedestrian crossing
prediction.

本研究提出了一种新颖的多模态预测算法，采用混合学习架构来分析环境的视觉特征和场景动态，预测行人的未来横穿行为，并在现有 2D 行人行为基准和新的 3D 驾驶数据集上进行评估，取得了最先进的表现。