In this paper, we study the challenging instance-wise vision-language tasks,
where the free-form language is required to align with the objects instead of
the whole image. To address these tasks, we propose X-DETR, whose architecture
has three major components: an object detector, a language encoder, and
vision-language alignment. The vision and language streams are independent
until the end and they are aligned using an efficient dot-product operation.
The whole network is trained end-to-end, such that the detector is optimized
for the vision-language tasks instead of an off-the-shelf component. To
overcome the limited size of paired object-language annotations, we leverage
other weak types of supervision to expand the knowledge coverage. This simple
yet effective architecture of X-DETR shows good accuracy and fast speeds for
multiple instance-wise vision-language tasks, e.g., 16.4 AP on LVIS detection
of 1.2K categories at ~20 frames per second without using any LVIS annotation
during training.

本文提出一种名为 X-DETR 的算法，针对具有挑战性的逐实例视觉语言任务进行研究；X-DETR 的架构包括对象检测器、语言编码器和视觉语言对齐器；该算法采用弱监督来扩大知识覆盖范围，经测试准确率高且速度快。