CLIPStyler demonstrated image style transfer with realistic textures using
only a style text description (instead of requiring a reference style image).
However, the ground semantics of objects in the style transfer output is lost
due to style spill-over on salient and background objects (content mismatch) or
over-stylization. To solve this, we propose Semantic CLIPStyler (Sem-CS), that
performs semantic style transfer. Sem-CS first segments the content image into
salient and non-salient objects and then transfers artistic style based on a
given style text description. The semantic style transfer is achieved using
global foreground loss (for salient objects) and global background loss (for
non-salient objects). Our empirical results, including DISTS, NIMA and user
study scores, show that our proposed framework yields superior qualitative and
quantitative performance. Our code is available at
github.com/chandagrover/sem-cs.

提出了一种基于全局前景损失和全局背景损失的 Semantic CLIPStyler 框架用于实现语义风格迁移，结果优于基于 CLIPStyler 的框架。

Sem-CS：面向基于文本的图像风格转换的语义 CLIP 样式器

Sem-CS: Semantic CLIPStyler for Text-Based Image Style Transfer

We address the new problem of language-guided semantic style transfer of 3D
indoor scenes. The input is a 3D indoor scene mesh and several phrases that
describe the target scene. Firstly, 3D vertex coordinates are mapped to RGB
residues by a multi-layer perceptron. Secondly, colored 3D meshes are
differentiablly rendered into 2D images, via a viewpoint sampling strategy
tailored for indoor scenes. Thirdly, rendered 2D images are compared to
phrases, via pre-trained vision-language models. Lastly, errors are
back-propagated to the multi-layer perceptron to update vertex colors
corresponding to certain semantic categories. We did large-scale qualitative
analyses and A/B user tests, with the public ScanNet and SceneNN datasets. We
demonstrate: (1) visually pleasing results that are potentially useful for
multimedia applications. (2) rendering 3D indoor scenes from viewpoints
consistent with human priors is important. (3) incorporating semantics
significantly improve style transfer quality. (4) an HSV regularization term
leads to results that are more consistent with inputs and generally rated
better. Codes and user study toolbox are available at
this https URL

本文介绍了一种语言引导的 3D 室内场景语义风格转移的方法，基于多层感知机，视觉语言模型，采用渲染和修正的方式实现创新，通过 ScanNet 数据集测试表明，语义信息的引入显著提高了风格转移的品质和用户评分。