Head pose estimation (HPE) task requires a sophisticated understanding of 3D
spatial relationships and precise numerical output of yaw, pitch, and roll
Euler angles. Previous HPE studies are mainly based on Non-large language
models (Non-LLMs), which rely on close-up human heads cropped from the full
image as inputs and lack robustness in real-world scenario. In this paper, we
present a novel framework to enhance the HPE prediction task by leveraging the
visual grounding capability of CogVLM. CogVLM is a vision language model (VLM)
with grounding capability of predicting object bounding boxes (BBoxes), which
enables HPE training and prediction using full image information input. To
integrate the HPE task into the VLM, we first cop with the catastrophic
forgetting problem in large language models (LLMs) by investigating the
rehearsal ratio in the data rehearsal method. Then, we propose and validate a
LoRA layer-based model merging method, which keeps the integrity of parameters,
to enhance the HPE performance in the framework. The results show our
HPE-CogVLM achieves a 31.5\% reduction in Mean Absolute Error for HPE
prediction over the current Non-LLM based state-of-the-art in cross-dataset
evaluation. Furthermore, we compare our LoRA layer-based model merging method
with LoRA fine-tuning only and other merging methods in CogVLM. The results
demonstrate our framework outperforms them in all HPE metrics.

本研究使用 CogVLM 的视觉定位能力，提出了一种新的框架来增强头部姿态估计任务，通过改进大语言模型中的灾难遗忘问题和引入 LoRA 层模型合并方法，有效提高头部姿态估计性能，并且在多个指标上优于现有方法。

HPE-CogVLM：基于视觉语言模型的新头部姿势定位任务探索

HPE-CogVLM: New Head Pose Grounding Task Exploration on Vision Language  Model

We introduce CogVLM, a powerful open-source visual language foundation model.
Different from the popular shallow alignment method which maps image features
into the input space of language model, CogVLM bridges the gap between the
frozen pretrained language model and image encoder by a trainable visual expert
module in the attention and FFN layers. As a result, CogVLM enables deep fusion
of vision language features without sacrificing any performance on NLP tasks.
CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal
benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+,
RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on
VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X
55B. Codes and checkpoints are available at this https URL

CogVLM 是一种强大的开源视觉语言基础模型，通过可训练的视觉专家模块在注意力和前馈网络层之间填补了预训练语言模型和图像编码器之间的差距，实现了视觉语言特征的深度融合，而不会牺牲在自然语言处理任务上的性能， 在 10 个经典的跨模态基准测试中取得了最先进的性能，包括 NoCaps，Flicker30k 字幕，RefCOCO，RefCOCO +，RefCOCOg，Visual7W，GQA，ScienceQA，VizWiz VQA 和 TDIUC， 在 VQAv2，OKVQA，TextVQA，COCO 字幕等方面排名第二，超过或与 PaLI-X 55B 相匹配。