Scene synthesis is a challenging problem with several industrial
applications. Recently, substantial efforts have been directed to synthesize
the scene using human motions, room layouts, or spatial graphs as the input.
However, few studies have addressed this problem from multiple modalities,
especially combining text prompts. In this paper, we propose a language-driven
scene synthesis task, which is a new task that integrates text prompts, human
motion, and existing objects for scene synthesis. Unlike other single-condition
synthesis tasks, our problem involves multiple conditions and requires a
strategy for processing and encoding them into a unified space. To address the
challenge, we present a multi-conditional diffusion model, which differs from
the implicit unification approach of other diffusion literature by explicitly
predicting the guiding points for the original data distribution. We
demonstrate that our approach is theoretically supportive. The intensive
experiment results illustrate that our method outperforms state-of-the-art
benchmarks and enables natural scene editing applications. The source code and
dataset can be accessed at this https URL

通过结合文字提示、人体动作和现有物体来进行多模态场景合成的多条件扩散模型，理论上支持，实验证明优于最先进的基准，并可实现自然场景编辑应用。

基于多条件扩散模型的语言驱动场景合成

Language-driven Scene Synthesis using Multi-conditional Diffusion Model

Segmentation of the infected areas of the lung is essential for quantifying
the severity of lung disease like pulmonary infections. Existing medical image
segmentation methods are almost uni-modal methods based on image. However,
these image-only methods tend to produce inaccurate results unless trained with
large amounts of annotated data. To overcome this challenge, we propose a
language-driven segmentation method that uses text prompt to improve to the
segmentation result. Experiments on the QaTa-COV19 dataset indicate that our
method improves the Dice score by 6.09% at least compared to the uni-modal
methods. Besides, our extended study reveals the flexibility of multi-modal
methods in terms of the information granularity of text and demonstrates that
multi-modal methods have a significant advantage over image-only methods in
terms of the size of training data required.

我们提出了一种语言驱动的分割方法，通过改善分割结果来量化肺部疾病的严重程度，这种方法在 QaTa-COV19 数据集上取得了显著的改进效果。

阿里亚德的线索：使用文本提示来改进胸部 X 射线图像中感染区域的分割

Ariadne's Thread:Using Text Prompts to Improve Segmentation of Infected  Areas from Chest X-ray images

Despite having promising results, style transfer, which requires preparing
style images in advance, may result in lack of creativity and accessibility.
Following human instruction, on the other hand, is the most natural way to
perform artistic style transfer that can significantly improve controllability
for visual effect applications. We introduce a new task, language-driven
artistic style transfer (LDAST), to manipulate the style of a content image,
guided by a text. We propose contrastive language visual artist (CLVA) that
learns to extract visual semantics from style instructions and accomplish LDAST
by the patch-wise style discriminator. The discriminator considers the
correlation between language and patches of style images or transferred results
to jointly embed style instructions. CLVA further compares contrastive pairs of
content images and style instructions to improve the mutual relativeness. The
results from the same content image can preserve consistent content structures.
Besides, they should present analogous style patterns from style instructions
that contain similar visual semantics. The experiments show that our CLVA is
effective and achieves superb transferred results on LDAST.

引入了一个新的任务，即以语言为驱动的艺术风格转移（LDAST），通过文本引导操纵内容图像的风格，所提出的对比语言视觉艺术家（CLVA）可以从风格指令中提取视觉语义并通过分块风格鉴别器实现 LDAST。

语言驱动的图像风格迁移

Language-Driven Image Style Transfer

Language-driven image editing can significantly save the laborious image
editing work and be friendly to the photography novice. However, most similar
work can only deal with a specific image domain or can only do global
retouching. To solve this new task, we first present a new language-driven
image editing dataset that supports both local and global editing with editing
operation and mask annotations. Besides, we also propose a baseline method that
fully utilizes the annotation to solve this problem. Our new method treats each
editing operation as a sub-module and can automatically predict operation
parameters. Not only performing well on challenging user data, but such an
approach is also highly interpretable. We believe our work, including both the
benchmark and the baseline, will advance the image editing area towards a more
general and free-form level.

本文介绍了一种利用语言驱动的图像编辑技术节省大量编辑工作量并对摄影新手友好，作者提出了一种支持本地和全局编辑的新技术方法，并构建了一个数据集用于支持本文所提出方法的研发，并且该方法在用户数据测试中表现良好，能够自动预测操作参数，具有高度的可解释性。