In autonomous driving, it is crucial to correctly interpret traffic gestures (TGs), such as those of an authority figure providing orders or instructions, or a pedestrian signaling the driver, to ensure a safe and pleasant traffic environment for all road users. This study investigates the capabilities of state-of-the-art vision-language models (VLMs) in zero-shot interpretation, focusing on their ability to caption and classify human gestures in traffic contexts. We create and publicly share two custom datasets with varying formal and informal TGs, such as 'Stop', 'Reverse', 'Hail', etc. The datasets are "Acted TG (ATG)" and "Instructive TG In-The-Wild (ITGI)". They are annotated with natural language, describing the pedestrian's body position and gesture. We evaluate models using three methods utilizing expert-generated captions as baseline and control: (1) caption similarity, (2) gesture classification, and (3) pose sequence reconstruction similarity. Results show that current VLMs struggle with gesture understanding: sentence similarity averages below 0.59, and classification F1 scores reach only 0.14-0.39, well below the expert baseline of 0.70. While pose reconstruction shows potential, it requires more data and refined metrics to be reliable. Our findings reveal that although some SOTA VLMs can interpret zero-shot human traffic gestures, none are accurate and robust enough to be trustworthy, emphasizing the need for further research in this domain.

本研究针对自动驾驶中交通手势的正确解读问题，探讨了当前先进的视觉语言模型(VLMs)在零样本情况下的理解能力。研究创建并公开了两个包含不同形式和 informal 手势的数据集，并发现现有模型在手势理解方面的准确性不足，强调了在该领域进一步研究的必要性。

Can Vision-Language Models Understand and Interpret Dynamic Gestures
  from Pedestrians? Pilot Datasets and Exploration Towards Instructive
  Nonverbal Commands for Cooperative Autonomous Vehicles

视觉语言模型是否能够理解和解释行人的动态手势？针对协作自动驾驶车辆的指令性非语言命令的初步数据集与探索