We propose a novel algorithm, named Open-Edit, which is the first attempt on
open-domain image manipulation with open-vocabulary instructions. It is a
challenging task considering the large variation of image domains and the lack
of training supervision. Our approach takes advantage of the unified
visual-semantic embedding space pretrained on a general image-caption dataset,
and manipulates the embedded visual features by applying text-guided vector
arithmetic on the image feature maps. A structure-preserving image decoder then
generates the manipulated images from the manipulated feature maps. We further
propose an on-the-fly sample-specific optimization approach with
cycle-consistency constraints to regularize the manipulated images and force
them to preserve details of the source images. Our approach shows promising
results in manipulating open-vocabulary color, texture, and high-level
attributes for various scenarios of open-domain images.

提出了 Open-Edit 算法，是一种处理开放域图像操作的新方法，采用基于文本图像翻译和生成的方式来操作图像，通过结构保持的图像解码器，调整图像特征映射来生成所需的操作图像。该方法在对开放词汇的颜色、纹理和高级特征进行处理方面取得了良好的结果。