In contemporary design practices, the integration of computer vision and
generative artificial intelligence (genAI) represents a transformative shift
towards more interactive and inclusive processes. These technologies offer new
dimensions of image analysis and generation, which are particularly relevant in
the context of urban landscape reconstruction. This paper presents a novel
workflow encapsulated within a prototype application, designed to leverage the
synergies between advanced image segmentation and diffusion models for a
comprehensive approach to urban design. Our methodology encompasses the
OneFormer model for detailed image segmentation and the Stable Diffusion XL
(SDXL) diffusion model, implemented through ControlNet, for generating images
from textual descriptions. Validation results indicated a high degree of
performance by the prototype application, showcasing significant accuracy in
both object detection and text-to-image generation. This was evidenced by
superior Intersection over Union (IoU) and CLIP scores across iterative
evaluations for various categories of urban landscape features. Preliminary
testing included utilising UrbanGenAI as an educational tool enhancing the
learning experience in design pedagogy, and as a participatory instrument
facilitating community-driven urban planning. Early results suggested that
UrbanGenAI not only advances the technical frontiers of urban landscape
reconstruction but also provides significant pedagogical and participatory
planning benefits. The ongoing development of UrbanGenAI aims to further
validate its effectiveness across broader contexts and integrate additional
features such as real-time feedback mechanisms and 3D modelling capabilities.
Keywords: generative AI; panoptic image segmentation; diffusion models; urban
landscape design; design pedagogy; co-design

这篇论文提出了一种集成计算机视觉和生成式人工智能（genAI）的新方法，通过先进的图像分割和扩散模型的协同作用，实现了城市设计的全面方法。我们的研究方法涵盖了详细图像分割的 OneFormer 模型和通过 ControlNet 实现的文本描述生成图像的稳定扩散 XL（SDXL）扩散模型。早期测试表明，UrbanGenAI 不仅提升了城市景观重建的技术前沿，也在设计教学和社区参与的城市规划方面提供了显著的益处。UrbanGenAI 的持续发展旨在对其在更广泛的背景下的有效性进行进一步验证，并集成实时反馈机制和三维建模能力等附加功能。