We propose FocusCLIP, integrating subject-level guidance--a specialized
mechanism for target-specific supervision--into the CLIP framework for improved
zero-shot transfer on human-centric tasks. Our novel contributions enhance CLIP
on both the vision and text sides. On the vision side, we incorporate ROI
heatmaps emulating human visual attention mechanisms to emphasize
subject-relevant image regions. On the text side, we introduce human pose
descriptions to provide rich contextual information. For human-centric tasks,
FocusCLIP is trained with images from the MPII Human Pose dataset. The proposed
approach surpassed CLIP by an average of 8.61% across five previously unseen
datasets covering three human-centric tasks. FocusCLIP achieved an average
accuracy of 33.65% compared to 25.04% by CLIP. We observed a 3.98% improvement
in activity recognition, a 14.78% improvement in age classification, and a
7.06% improvement in emotion recognition. Moreover, using our proposed
single-shot LLM prompting strategy, we release a high-quality MPII Pose
Descriptions dataset to encourage further research in multimodal learning for
human-centric tasks. Furthermore, we also demonstrate the effectiveness of our
subject-level supervision on non-human-centric tasks. FocusCLIP shows a 2.47%
improvement over CLIP in zero-shot bird classification using the CUB dataset.
Our findings emphasize the potential of integrating subject-level guidance with
general pretraining methods for enhanced downstream performance.

我们提出了 FocusCLIP，在 CLIP 框架中整合了主题级引导 —— 一种针对特定目标的监督机制，从而在人类中心任务的零样本迁移方面取得了改进。

FocusCLIP: 人类中心任务中的多模态主体级指导零样本迁移

FocusCLIP: Multimodal Subject-Level Guidance for Zero-Shot Transfer in  Human-Centric Tasks

Large Language Models (LLMs) have demonstrated remarkable generalization
across diverse tasks, leading individuals to increasingly use them as personal
assistants and universal computing engines. Nevertheless, a notable obstacle
emerges when feeding numerical/temporal data into these models, such as data
sourced from wearables or electronic health records. LLMs employ tokenizers in
their input that break down text into smaller units. However, tokenizers are
not designed to represent numerical values and might struggle to understand
repetitive patterns and context, treating consecutive values as separate tokens
and disregarding their temporal relationships. Here, we discuss recent works
that employ LLMs for human-centric tasks such as in mobile health sensing and
present a case study showing that popular LLMs tokenize temporal data
incorrectly. To address that, we highlight potential solutions such as prompt
tuning with lightweight embedding layers as well as multimodal adapters, that
can help bridge this "modality gap". While the capability of language models to
generalize to other modalities with minimal or no finetuning is exciting, this
paper underscores the fact that their outputs cannot be meaningful if they
stumble over input nuances.

介绍了大型语言模型（LLMs）在移动健康感知等人类中心任务中的使用，并通过案例研究证明了目前流行的 LLMs 对于时间数据的分词处理存在错误，提出了解决方案如轻量级嵌入层调整和多模态适配器来弥合模态差异。该论文强调了语言模型在处理输入细微差别时无法产生有意义的输出。