The capacity of existing human keypoint localization models is limited by
keypoint priors provided by the training data. To alleviate this restriction
and pursue more general model, this work studies keypoint localization from a
different perspective by reasoning locations based on keypiont clues in text
descriptions. We propose LocLLM, the first Large-Language Model (LLM) based
keypoint localization model that takes images and text instructions as inputs
and outputs the desired keypoint coordinates. LocLLM leverages the strong
reasoning capability of LLM and clues of keypoint type, location, and
relationship in textual descriptions for keypoint localization. To effectively
tune LocLLM, we construct localization-based instruction conversations to
connect keypoint description with corresponding coordinates in input image, and
fine-tune the whole model in a parameter-efficient training pipeline. LocLLM
shows remarkable performance on standard 2D/3D keypoint localization
benchmarks. Moreover, incorporating language clues into the localization makes
LocLLM show superior flexibility and generalizable capability in cross dataset
keypoint localization, and even detecting novel type of keypoints unseen during
training.

通过使用文本描述中的关键点线索，借助 LocLLM，将大规模语言模型（LLM）与图像和文本指令作为输入，输出所需的关键点坐标，有效扩展了现有的关键点定位模型的能力。

LocLLM: 大型语言模型通过人体关键点定位实现通用化

LocLLM: Exploiting Generalizable Human Keypoint Localization via Large  Language Model

Retrieval Augmented Generation (RAG) systems have seen huge popularity in
augmenting Large-Language Model (LLM) outputs with domain specific and time
sensitive data. Very recently a shift is happening from simple RAG setups that
query a vector database for additional information with every user input to
more sophisticated forms of RAG. However, different concrete approaches compete
on mostly anecdotal evidence at the moment. In this paper we present a rigorous
dataset creation and evaluation workflow to quantitatively compare different
RAG strategies. We use a dataset created this way for the development and
evaluation of a boolean agent RAG setup: A system in which a LLM can decide
whether to query a vector database or not, thus saving tokens on questions that
can be answered with internal knowledge. We publish our code and generated
dataset online.

检索增强生成（RAG）系统在通过领域特定和时间敏感数据增强大型语言模型（LLM）输出方面非常受欢迎。本文提出了一种严格的数据集创建和评估工作流程，从而定量比较了不同的 RAG 策略，用于布尔代理 RAG 设置的开发和评估。我们在线发布了我们的代码和生成的数据集。