Current instruction-based editing methods, such as InstructPix2Pix, often
fail to produce satisfactory results in complex scenarios due to their
dependence on the simple CLIP text encoder in diffusion models. To rectify
this, this paper introduces SmartEdit, a novel approach to instruction-based
image editing that leverages Multimodal Large Language Models (MLLMs) to
enhance their understanding and reasoning capabilities. However, direct
integration of these elements still faces challenges in situations requiring
complex reasoning. To mitigate this, we propose a Bidirectional Interaction
Module that enables comprehensive bidirectional information interactions
between the input image and the MLLM output. During training, we initially
incorporate perception data to boost the perception and understanding
capabilities of diffusion models. Subsequently, we demonstrate that a small
amount of complex instruction editing data can effectively stimulate
SmartEdit's editing capabilities for more complex instructions. We further
construct a new evaluation dataset, Reason-Edit, specifically tailored for
complex instruction-based image editing. Both quantitative and qualitative
results on this evaluation dataset indicate that our SmartEdit surpasses
previous methods, paving the way for the practical application of complex
instruction-based image editing.

SmartEdit 是一种基于指令的图像编辑方法，利用多模态大型语言模型（MLLMS）增强理解和推理能力，并通过双向交互模块实现输入图像和 MLLM 输出之间的综合双向信息交互，从而实现更复杂指令下的图像编辑。通过新构建的 Reason-Edit 评估数据集，定量和定性结果表明 SmartEdit 优于先前的方法，为复杂指令 - based 图像编辑的实际应用铺平了道路。