Recently, researchers have proposed powerful systems for generating and
manipulating images using natural language instructions. However, it is
difficult to precisely specify many common classes of image transformations
with text alone. For example, a user may wish to change the location and breed
of a particular dog in an image with several similar dogs. This task is quite
difficult with natural language alone, and would require a user to write a
laboriously complex prompt that both disambiguates the target dog and describes
the destination. We propose ClickDiffusion, a system for precise image
manipulation and generation that combines natural language instructions with
visual feedback provided by the user through a direct manipulation interface.
We demonstrate that by serializing both an image and a multi-modal instruction
into a textual representation it is possible to leverage LLMs to perform
precise transformations of the layout and appearance of an image. Code
available at this https URL

提出了一种名为 ClickDiffusion 的系统，结合自然语言指令和用户通过直接操作界面提供的可视化反馈，将图像和多模态指令序列化为文本表示形式，以实现图像的精确变换和生成。