Human-centric perception tasks, e.g., human mesh recovery, pedestrian
detection, skeleton-based action recognition, and pose estimation, have wide
industrial applications, such as metaverse and sports analysis. There is a
recent surge to develop human-centric foundation models that can benefit a
broad range of human-centric perception tasks. While many human-centric
foundation models have achieved success, most of them only excel in 2D vision
tasks or require extensive fine-tuning for practical deployment in real-world
scenarios. These limitations severely restrict their usability across various
downstream tasks and situations. To tackle these problems, we present Hulk, the
first multimodal human-centric generalist model, capable of addressing most of
the mainstream tasks simultaneously without task-specific finetuning, covering
2D vision, 3D vision, skeleton-based, and vision-language tasks. The key to
achieving this is condensing various task-specific heads into two general
heads, one for discrete representations, e.g., languages, and the other for
continuous representations, e.g., location coordinates. The outputs of two
heads can be further stacked into four distinct input and output modalities.
This uniform representation enables Hulk to treat human-centric tasks as
modality translation, integrating knowledge across a wide range of tasks. To
validate the effectiveness of our proposed method, we conduct comprehensive
experiments on 11 benchmarks across 8 human-centric tasks. Experimental results
surpass previous methods substantially, demonstrating the superiority of our
proposed method. The code will be available on
this https URL

Hulk 是第一个多模态的人本主义通用模型，可以同时处理大多数主流任务，无需任务特定的微调，涵盖 2D 视觉、3D 视觉、基于骨架的和视觉语言任务。