Numerous methods have been proposed to detect, estimate, and analyze
properties of people in images, including the estimation of 3D pose, shape,
contact, human-object interaction, emotion, and more. Each of these methods
works in isolation instead of synergistically. Here we address this problem and
build a language-driven human understanding system -- ChatHuman, which combines
and integrates the skills of many different methods. To do so, we finetune a
Large Language Model (LLM) to select and use a wide variety of existing tools
in response to user inputs. In doing so, ChatHuman is able to combine
information from multiple tools to solve problems more accurately than the
individual tools themselves and to leverage tool output to improve its ability
to reason about humans. The novel features of ChatHuman include leveraging
academic publications to guide the application of 3D human-related tools,
employing a retrieval-augmented generation model to generate
in-context-learning examples for handling new tools, and discriminating and
integrating tool results to enhance 3D human understanding. Our experiments
show that ChatHuman outperforms existing models in both tool selection accuracy
and performance across multiple 3D human-related tasks. ChatHuman is a step
towards consolidating diverse methods for human analysis into a single,
powerful, system for 3D human reasoning.

ChatHuman 是一个基于语言驱动的人类理解系统，通过结合和整合多种不同方法的技能，利用大型语言模型（LLM）选择和使用各种现有工具来解决 3D 人类相关问题，并通过学术出版物指导应用 3D 人类相关工具、生成上下文学习示例以处理新工具以及辨别和整合工具结果以提升对 3D 人类理解的能力。与现有模型相比，在工具选择准确性和多个 3D 人类相关任务的性能方面，ChatHuman 表现出更好的性能，是将多样方法整合为单一强大的 3D 人类推理系统的一步。