The core problem in zero-shot open vocabulary detection is how to align
visual and text features, so that the detector performs well on unseen classes.
Previous approaches train the feature pyramid and detection head from scratch,
which breaks the vision-text feature alignment established during pretraining,
and struggles to prevent the language model from forgetting unseen classes.
We propose three methods to alleviate these issues. Firstly, a simple scheme
is used to augment the text embeddings which prevents overfitting to a small
number of classes seen during training, while simultaneously saving memory and
computation. Secondly, the feature pyramid network and the detection head are
modified to include trainable gated shortcuts, which encourages vision-text
feature alignment and guarantees it at the start of detection training.
Finally, a self-training approach is used to leverage a larger corpus of
image-text pairs thus improving detection performance on classes with no human
annotated bounding boxes.
Our three methods are evaluated on the zero-shot version of the LVIS
benchmark, each of them showing clear and significant benefits. Our final
network achieves the new stateof-the-art on the mAP-all metric and demonstrates
competitive performance for mAP-rare, as well as superior transfer to COCO and
Objects365.

提出三种方法来解决零样本开放词汇检测中的视觉和文本特征对齐问题，其中包括改进的特征金字塔网络和检测头，自我训练方法以及使用更大的图像文本对语料库来改善检测性能，并在 LVIS 基准测试上获得了新的最优性能。

提高开放词汇检测的特征对齐的三种方法

Three ways to improve feature alignment for open vocabulary detection

Recently proposed decoupled training methods emerge as a dominant paradigm
for long-tailed object detection. But they require an extra fine-tuning stage,
and the disjointed optimization of representation and classifier might lead to
suboptimal results. However, end-to-end training methods, like equalization
loss (EQL), still perform worse than decoupled training methods. In this paper,
we reveal the main issue in long-tailed object detection is the imbalanced
gradients between positives and negatives, and find that EQL does not solve it
well. To address the problem of imbalanced gradients, we introduce a new
version of equalization loss, called equalization loss v2 (EQL v2), a novel
gradient guided reweighing mechanism that re-balances the training process for
each category independently and equally. Extensive experiments are performed on
the challenging LVIS benchmark. EQL v2 outperforms origin EQL by about 4 points
overall AP with 14-18 points improvements on the rare categories. More
importantly, it also surpasses decoupled training methods. Without further
tuning for the Open Images dataset, EQL v2 improves EQL by 7.3 points AP,
showing strong generalization ability. Codes have been released at
this https URL

为解决不平衡梯度导致的长尾目标检测困难，本研究提出了一种新的均衡损失（EQL v2）。实验结果表明，EQL v2 在 LVIS 基准测试中表现优异，比起原 EQL 可提高 4 个点的总 AP 值和 14-18 个点的罕见类别 AP 值，并且甚至超过了分离式训练方法。