Recent years have witnessed the rapid progress of image captioning. However,
the demands for large memory storage and heavy computational burden prevent
these captioning models from being deployed on mobile devices. The main
obstacles lie in the heavyweight visual feature extractors (i.e., object
detectors) and complicated cross-modal fusion networks. To this end, we propose
LightCap, a lightweight image captioner for resource-limited devices. The core
design is built on the recent CLIP model for efficient image captioning. To be
specific, on the one hand, we leverage the CLIP model to extract the compact
grid features without relying on the time-consuming object detectors. On the
other hand, we transfer the image-text retrieval design of CLIP to image
captioning scenarios by devising a novel visual concept extractor and a
cross-modal modulator. We further optimize the cross-modal fusion model and
parallel prediction heads via sequential and ensemble distillations. With the
carefully designed architecture, our model merely contains 40M parameters,
saving the model size by more than 75% and the FLOPs by more than 98% in
comparison with the current state-of-the-art methods. In spite of the low
capacity, our model still exhibits state-of-the-art performance on prevalent
datasets, e.g., 136.6 CIDEr on COCO Karpathy test split. Testing on the
smartphone with only a single CPU, the proposed LightCap exhibits a fast
inference speed of 188ms per image, which is ready for practical applications.

提出了一种基于 CLIP 模型的轻量级图像字幕生成器 LightCap，可以在资源有限的设备上使用，仅包含 40M 参数，比现有技术方法的模型大小减少了超过 75％，与仅使用单个 CPU 的手机的推理速度为 188ms 每张图像，并在著名数据集上表现出最先进的性能。