The rapid development of diffusion models has triggered diverse applications.
Identity-preserving text-to-image generation (ID-T2I) particularly has received
significant attention due to its wide range of application scenarios like AI
portrait and advertising. While existing ID-T2I methods have demonstrated
impressive results, several key challenges remain: (1) It is hard to maintain
the identity characteristics of reference portraits accurately, (2) The
generated images lack aesthetic appeal especially while enforcing identity
retention, and (3) There is a limitation that cannot be compatible with
LoRA-based and Adapter-based methods simultaneously. To address these issues,
we present \textbf{ID-Aligner}, a general feedback learning framework to
enhance ID-T2I performance. To resolve identity features lost, we introduce
identity consistency reward fine-tuning to utilize the feedback from face
detection and recognition models to improve generated identity preservation.
Furthermore, we propose identity aesthetic reward fine-tuning leveraging
rewards from human-annotated preference data and automatically constructed
feedback on character structure generation to provide aesthetic tuning signals.
Thanks to its universal feedback fine-tuning framework, our method can be
readily applied to both LoRA and Adapter models, achieving consistent
performance gains. Extensive experiments on SD1.5 and SDXL diffusion models
validate the effectiveness of our approach. \textbf{Project Page:
https://idaligner.github.io/}

通过引入通用反馈学习框架，本研究提出了 ID-Aligner 方法用于改进 ID-T2I 性能，解决参考肖像的身份特征保留、生成图像的美学吸引力缺失以及与 LoRA 和 Adapter 方法的兼容性等关键问题。大量的实验证明了这一方法的有效性。

ID-Aligner: 用奖励反馈学习增强保持身份的图文生成

ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with  Reward Feedback Learning

Given a source portrait, the automatic human body reshaping task aims at
editing it to an aesthetic body shape. As the technology has been widely used
in media, several methods have been proposed mainly focusing on generating
optical flow to warp the body shape. However, those previous works only
consider the local transformation of different body parts (arms, torso, and
legs), ignoring the global affinity, and limiting the capacity to ensure
consistency and quality across the entire body. In this paper, we propose a
novel Adaptive Affinity-Graph Network (AAGN), which extracts the global
affinity between different body parts to enhance the quality of the generated
optical flow. Specifically, our AAGN primarily introduces the following
designs: (1) we propose an Adaptive Affinity-Graph (AAG) Block that leverages
the characteristic of a fully connected graph. AAG represents different body
parts as nodes in an adaptive fully connected graph and captures all the
affinities between nodes to obtain a global affinity map. The design could
better improve the consistency between body parts. (2) Besides, for
high-frequency details are crucial for photo aesthetics, a Body Shape
Discriminator (BSD) is designed to extract information from both high-frequency
and spatial domain. Particularly, an SRM filter is utilized to extract
high-frequency details, which are combined with spatial features as input to
the BSD. With this design, BSD guides the Flow Generator (FG) to pay attention
to various fine details rather than rigid pixel-level fitting. Extensive
experiments conducted on the BR-5K dataset demonstrate that our framework
significantly enhances the aesthetic appeal of reshaped photos, marginally
surpassing all previous work to achieve state-of-the-art in all evaluation
metrics.

通过引入自适应亲和 - 图网络，本研究旨在通过生成光流对身体形状中的局部和全局转换进行编辑，从而提升重新塑造照片的美感，超越之前的工作，并在所有评估指标上实现最先进的效果。

基于自适应亲和图网络的结构感知人体重塑

Structure-Aware Human Body Reshaping with Adaptive Affinity-Graph  Network

Despite their wide-spread success, Text-to-Image models (T2I) still struggle
to produce images that are both aesthetically pleasing and faithful to the
user's input text. We introduce DreamSync, a model-agnostic training algorithm
by design that improves T2I models to be faithful to the text input. DreamSync
builds off a recent insight from TIFA's evaluation framework -- that large
vision-language models (VLMs) can effectively identify the fine-grained
discrepancies between generated images and the text inputs. DreamSync uses this
insight to train T2I models without any labeled data; it improves T2I models
using its own generations. First, it prompts the model to generate several
candidate images for a given input text. Then, it uses two VLMs to select the
best generation: a Visual Question Answering model that measures the alignment
of generated images to the text, and another that measures the generation's
aesthetic quality. After selection, we use LoRA to iteratively finetune the T2I
model to guide its generation towards the selected best generations. DreamSync
does not need any additional human annotation. model architecture changes, or
reinforcement learning. Despite its simplicity, DreamSync improves both the
semantic alignment and aesthetic appeal of two diffusion-based T2I models,
evidenced by multiple benchmarks (+1.7% on TIFA, +2.9% on DSG1K, +3.4% on VILA
aesthetic) and human evaluation.

通过使用无需标记数据的 DreamSync 训练算法，结合大视觉语言模型（VLMs）来改进文本到图像模型（T2I）的语义对齐和美观度。

DreamSync：将文本到图像生成与图像理解反馈对齐

DreamSync: Aligning Text-to-Image Generation with Image Understanding  Feedback

Text-to-image generation has recently witnessed remarkable achievements. We
introduce a text-conditional image diffusion model, termed RAPHAEL, to generate
highly artistic images, which accurately portray the text prompts, encompassing
multiple nouns, adjectives, and verbs. This is achieved by stacking tens of
mixture-of-experts (MoEs) layers, i.e., space-MoE and time-MoE layers, enabling
billions of diffusion paths (routes) from the network input to the output. Each
path intuitively functions as a "painter" for depicting a particular textual
concept onto a specified image region at a diffusion timestep. Comprehensive
experiments reveal that RAPHAEL outperforms recent cutting-edge models, such as
Stable Diffusion, ERNIE-ViLG 2.0, DeepFloyd, and DALL-E 2, in terms of both
image quality and aesthetic appeal. Firstly, RAPHAEL exhibits superior
performance in switching images across diverse styles, such as Japanese comics,
realism, cyberpunk, and ink illustration. Secondly, a single model with three
billion parameters, trained on 1,000 A100 GPUs for two months, achieves a
state-of-the-art zero-shot FID score of 6.61 on the COCO dataset. Furthermore,
RAPHAEL significantly surpasses its counterparts in human evaluation on the
ViLG-300 benchmark. We believe that RAPHAEL holds the potential to propel the
frontiers of image generation research in both academia and industry, paving
the way for future breakthroughs in this rapidly evolving field. More details
can be found on a project webpage: this https URL

本文介绍了一种基于 Mixture-of-Experts 的文本条件化图像扩散模型（RAPHAEL），其在图像质量和美感上优于现有的一些先进模型（包括 Stable Diffusion、ERNIE-ViLG 2.0、DeepFloyd 和 DALL-E 2）