Text-to-video (T2V) models have shown remarkable capabilities in generating
diverse videos. However, they struggle to produce user-desired stylized videos
due to (i) text's inherent clumsiness in expressing specific styles and (ii)
the generally degraded style fidelity. To address these challenges, we
introduce StyleCrafter, a generic method that enhances pre-trained T2V models
with a style control adapter, enabling video generation in any style by
providing a reference image. Considering the scarcity of stylized video
datasets, we propose to first train a style control adapter using style-rich
image datasets, then transfer the learned stylization ability to video
generation through a tailor-made finetuning paradigm. To promote content-style
disentanglement, we remove style descriptions from the text prompt and extract
style information solely from the reference image using a decoupling learning
strategy. Additionally, we design a scale-adaptive fusion module to balance the
influences of text-based content features and image-based style features, which
helps generalization across various text and style combinations. StyleCrafter
efficiently generates high-quality stylized videos that align with the content
of the texts and resemble the style of the reference images. Experiments
demonstrate that our approach is more flexible and efficient than existing
competitors.

通过引入 StyleCrafter 方法，扩展预训练的 T2V 模型，实现风格控制适配器，通过提供参考图像在任何风格下生成视频。通过首先使用富有风格的图像数据集训练一个风格控制适配器，然后通过独立的学习策略从参考图像中提取风格信息，在文字提示中去除风格描述，实现内容和风格的分离，通过一个自适应融合模块平衡文本内容特征和图像风格特征的影响，StyleCrafter 能够高效生成与文本内容一致且符合参考图像风格的高质量风格化视频，并且实验证明我们的方法比现有竞争对手更灵活和高效。

StyleCrafter：通过风格适配器提升风格化文本到视频生成

StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style  Adapter

Generating a new font library is a very labor-intensive and time-consuming
job for glyph-rich scripts. Few-shot font generation is thus required, as it
requires only a few glyph references without fine-tuning during test. Existing
methods follow the style-content disentanglement paradigm and expect novel
fonts to be produced by combining the style codes of the reference glyphs and
the content representations of the source. However, these few-shot font
generation methods either fail to capture content-independent style
representations, or employ localized component-wise style representations,
which is insufficient to model many Chinese font styles that involve
hyper-component features such as inter-component spacing and
"connected-stroke". To resolve these drawbacks and make the style
representations more reliable, we propose a self-supervised cross-modality
pre-training strategy and a cross-modality transformer-based encoder that is
conditioned jointly on the glyph image and the corresponding stroke labels. The
cross-modality encoder is pre-trained in a self-supervised manner to allow
effective capture of cross- and intra-modality correlations, which facilitates
the content-style disentanglement and modeling style representations of all
scales (stroke-level, component-level and character-level). The pre-trained
encoder is then applied to the downstream font generation task without
fine-tuning. Experimental comparisons of our method with state-of-the-art
methods demonstrate our method successfully transfers styles of all scales. In
addition, it only requires one reference glyph and achieves the lowest rate of
bad cases in the few-shot font generation task 28% lower than the second best

该研究提出了一种自我监督的跨模态预训练策略和基于跨模态 Transformer 的编码器，用于少样本字体生成，并成功实现了各种规模风格的转移，只需要一个参考字形且错误率最低。

XMP-Font: 自监督跨模态预训练 Few-shot 字体生成

XMP-Font: Self-Supervised Cross-Modality Pre-training for Few-Shot Font  Generation

A few-shot font generation (FFG) method has to satisfy two objectives: the
generated images should preserve the underlying global structure of the target
character and present the diverse local reference style. Existing FFG methods
aim to disentangle content and style either by extracting a universal
representation style or extracting multiple component-wise style
representations. However, previous methods either fail to capture diverse local
styles or cannot be generalized to a character with unseen components, e.g.,
unseen language systems. To mitigate the issues, we propose a novel FFG method,
named Multiple Localized Experts Few-shot Font Generation Network (MX-Font).
MX-Font extracts multiple style features not explicitly conditioned on
component labels, but automatically by multiple experts to represent different
local concepts, e.g., left-side sub-glyph. Owing to the multiple experts,
MX-Font can capture diverse local concepts and show the generalizability to
unseen languages. During training, we utilize component labels as weak
supervision to guide each expert to be specialized for different local
concepts. We formulate the component assign problem to each expert as the graph
matching problem, and solve it by the Hungarian algorithm. We also employ the
independence loss and the content-style adversarial loss to impose the
content-style disentanglement. In our experiments, MX-Font outperforms previous
state-of-the-art FFG methods in the Chinese generation and cross-lingual, e.g.,
Chinese to Korean, generation. Source code is available at
this https URL

本论文提出一种名为 MX-Font 的新颖的 Few-shot 字体生成方法，使用多个专家提取多个特征来表示不同的本地概念，解决现有方法不能捕捉多样化当地的风格或不具有泛化到未知组件的字符的问题，这在中文和跨语言生成方面表现出色。