Text spotting, a task involving the extraction of textual information from
image or video sequences, faces challenges in cross-domain adaption, such as
image-to-image and image-to-video generalization. In this paper, we introduce a
new method, termed VimTS, which enhances the generalization ability of the
model by achieving better synergy among different tasks. Typically, we propose
a Prompt Queries Generation Module and a Tasks-aware Adapter to effectively
convert the original single-task model into a multi-task model suitable for
both image and video scenarios with minimal additional parameters. The Prompt
Queries Generation Module facilitates explicit interaction between different
tasks, while the Tasks-aware Adapter helps the model dynamically learn suitable
features for each task. Additionally, to further enable the model to learn
temporal information at a lower cost, we propose a synthetic video text dataset
(VTD-368k) by leveraging the Content Deformation Fields (CoDeF) algorithm.
Notably, our method outperforms the state-of-the-art method by an average of
2.6% in six cross-domain benchmarks such as TT-to-IC15, CTW1500-to-TT, and
TT-to-CTW1500. For video-level cross-domain adaption, our method even surpasses
the previous end-to-end video spotting method in ICDAR2015 video and DSText v2
by an average of 5.5% on the MOTA metric, using only image-level data. We
further demonstrate that existing Large Multimodal Models exhibit limitations
in generating cross-domain scene text spotting, in contrast to our VimTS model
which requires significantly fewer parameters and data. The code and datasets
will be made available at the this https URL

通过集成多任务模型、合成视频文本数据集和学习时序信息，VimTS 模型在跨领域场景文本检测任务中展现出卓越的性能，并且相较于现有的大型多模态模型，所需参数和数据更为少。

VimTS：用于增强跨领域泛化能力的统一视频和图像文本识别器

VimTS: A Unified Video and Image Text Spotter for Enhancing the  Cross-domain Generalization

In transfer learning, only the last part of the networks - the so-called head
- is often fine-tuned. Representation similarity analysis shows that the most
significant change still occurs in the head even if all weights are updatable.
However, recent results from few-shot learning have shown that representation
change in the early layers, which are mostly convolutional, is beneficial,
especially in the case of cross-domain adaption. In our paper, we find out
whether that also holds true for transfer learning. In addition, we analyze the
change of representation in transfer learning, both during pre-training and
fine-tuning, and find out that pre-trained structure is unlearned if not
usable.

本文研究转移学习中的表示变化，并发现在少样本学习中，在转移学习的早期层次中进行表示变化是有益的，特别是在跨域适应的情况下。同时，我们发现在转移学习的预训练和微调过程中，预训练结构如果不能使用，则不能被学习。

预训练和微调表示的相似性

Similarity of Pre-trained and Fine-tuned Representations

How does language differ across one's Facebook status updates vs. one's text
messages (SMS)? In this study, we show how Facebook and SMS use differs in
psycho-linguistic characteristics and how these differences drive downstream
analyses with an illustration of depression diagnosis. We use a sample of
consenting participants who shared Facebook status updates, SMS data, and
answered a standard psychological depression screener. We quantify domain
differences using psychologically driven lexical methods and find that language
on Facebook involves more personal concerns, experiences, and content features
while the language in SMS contains more informal and style features. Next, we
estimate depression from both text domains, using a depression model trained on
Facebook data, and find a drop in accuracy when predicting self-reported
depression assessments from the SMS-based depression estimates. Finally, we
evaluate a simple domain adaption correction based on words driving the
cross-platform differences and applied it to the SMS-derived depression
estimates, resulting in significant improvement in prediction. Our work shows
the Facebook vs. SMS difference in language use and suggests the necessity of
cross-domain adaption for text-based predictions.

通过对 Facebook 和短消息 (SMS) 的语言差异进行心理语言学特征研究，本研究通过对抑郁症诊断的说明展示了这些差异如何推动下游分析。我们使用了一个共同允许分享 Facebook 状态更新、短消息数据外加填写标准心理抑郁症检测表的样本。我们使用基于心理学的词汇方法量化域之间的差异，并发现 Facebook 上的语言涉及更多的个人关注、经历和内容特征，而短信中的语言则包含更多非正式和风格特征。最后，我们评估了一个简单的领域适应纠正算法，应用于基于短信的抑郁症估计，使预测指标有了显著的提高，强调了跨领域适应对于基于文本的预测的必要性。