Supervised image captioning approaches have made great progress, but it is
challenging to collect high-quality human-annotated image-text data. Recently,
large-scale vision and language models (e.g., CLIP) and large-scale generative
language models (e.g., GPT-2) have shown strong performances in various tasks,
which also provide some new solutions for image captioning with web paired
data, unpaired data or even text-only data. Among them, the mainstream solution
is to project image embeddings into the text embedding space with the
assistance of consistent representations between image-text pairs from the CLIP
model. However, the current methods still face several challenges in adapting
to the diversity of data configurations in a unified solution, accurately
estimating image-text embedding bias, and correcting unsatisfactory prediction
results in the inference stage. This paper proposes a new Text data-centric
approach with Interactive Prompts for image Captioning, named TIPCap. 1) We
consider four different settings which gradually reduce the dependence on
paired data. 2) We construct a mapping module driven by multivariate Gaussian
distribution to mitigate the modality gap, which is applicable to the above
four different settings. 3) We propose a prompt interaction module that can
incorporate optional prompt information before generating captions. Extensive
experiments show that our TIPCap outperforms other weakly or unsupervised image
captioning methods and achieves a new state-of-the-art performance on two
widely used datasets, i.e., MS-COCO and Flickr30K.

TIPCap 是一种基于文本数据的交互提示的图像标题生成方法，通过减少对配对数据的依赖性、构建驱动多变量高斯分布的映射模块以缓解模态差异，并在生成标题之前引入可选的提示信息，优于其他弱监督或无监督的图像标题生成方法，并在 MS-COCO 和 Flickr30K 等常用数据集上达到了最新的最优性能。

基于文本数据的图像标题生成与交互提示

Text Data-Centric Image Captioning with Interactive Prompts

Personalization of speech models on mobile devices (on-device
personalization) is an active area of research, but more often than not, mobile
devices have more text-only data than paired audio-text data. We explore
training a personalized language model on text-only data, used during inference
to improve speech recognition performance for that user. We experiment on a
user-clustered LibriSpeech corpus, supplemented with personalized text-only
data for each user from Project Gutenberg. We release this User-Specific
LibriSpeech (UserLibri) dataset to aid future personalization research.
LibriSpeech audio-transcript pairs are grouped into 55 users from the
test-clean dataset and 52 users from test-other. We are able to lower the
average word error rate per user across both sets in streaming and nonstreaming
models, including an improvement of 2.5 for the harder set of test-other users
when streaming.

通过在移动设备上使用文本数据进行语言模型的个性化，提高了针对该用户的语音识别性能。利用 LibriSpeech 语料库和 Project Gutenberg 的个性化文本，我们对 55 个用户的文字 - 语音配对数据进行了实验，并提供了 UserLibri 数据集以帮助未来个性化研究。实验结果表明，在流媒体和非流媒体模型中，我们能够降低每个用户的平均单词错误率，其中在流媒体情况下，难度较大的测试集错误率下降了 2.5%