This report introduces an enhanced method for the Foundational Few-Shot
Object Detection (FSOD) task, leveraging the vision-language model (VLM) for
object detection. However, on specific datasets, VLM may encounter the problem
where the detected targets are misaligned with the target concepts of interest.
This misalignment hinders the zero-shot performance of VLM and the application
of fine-tuning methods based on pseudo-labels. To address this issue, we
propose the VLM+ framework, which integrates the multimodal large language
model (MM-LLM). Specifically, we use MM-LLM to generate a series of referential
expressions for each category. Based on the VLM predictions and the given
annotations, we select the best referential expression for each category by
matching the maximum IoU. Subsequently, we use these referential expressions to
generate pseudo-labels for all images in the training set and then combine them
with the original labeled data to fine-tune the VLM. Additionally, we employ
iterative pseudo-label generation and optimization to further enhance the
performance of the VLM. Our approach achieve 32.56 mAP in the final test.

本报告介绍了一种增强的方法来解决基础性少样本物体检测任务，利用视觉语言模型 (VLM) 进行物体检测。为解决 VLM 可能遇到的与目标概念不一致的问题，本研究提出了 VLM + 框架，把多模态大型语言模型 (MM-LLM) 集成到 VLM 中。通过 MM-LLM 生成每个类别的一系列参考表达式，根据 VLM 的预测和给定的注释，选取与每个类别最大 IoU 匹配的最佳参考表达式，并生成伪标签来细调 VLM。该方法在最终测试中达到了 32.56 mAP。