Recently, large multimodal models have built a bridge from visual to textual
information, but they tend to underperform in remote sensing scenarios. This
underperformance is due to the complex distribution of objects and the
significant scale differences among targets in remote sensing images, leading
to visual ambiguities and insufficient descriptions by these multimodal models.
Moreover, the lack of multimodal fine-tuning data specific to the remote
sensing field makes it challenging for the model's behavior to align with user
queries. To address these issues, this paper proposes an attribute-guided
\textbf{Multi-Granularity Instruction Multimodal Model (MGIMM)} for remote
sensing image detailed description. MGIMM guides the multimodal model to learn
the consistency between visual regions and corresponding text attributes (such
as object names, colors, and shapes) through region-level instruction tuning.
Then, with the multimodal model aligned on region-attribute, guided by
multi-grain visual features, MGIMM fully perceives both region-level and global
image information, utilizing large language models for comprehensive
descriptions of remote sensing images. Due to the lack of a standard benchmark
for generating detailed descriptions of remote sensing images, we construct a
dataset featuring 38,320 region-attribute pairs and 23,463 image-detailed
description pairs. Compared with various advanced methods on this dataset, the
results demonstrate the effectiveness of MGIMM's region-attribute guided
learning approach. Code can be available at
this https URL

本文提出了一种属性引导的多粒度指令多模型（MGIMM），用于遥感图像的详细描述，并通过在区域级别指导调整实现了多模型的一致性学习。通过多颗粒度的视觉特征，MGIMM 可以充分感知区域级和全局图像信息，利用大语言模型对遥感图像进行全面的描述。实验证明了 MGIMM 的区域 - 属性引导学习方法的有效性。

MGIMM：面向属性引导的遥感影像细节描述的多粒度指令多模态模型

MGIMM: Multi-Granularity Instruction Multimodal Model for  Attribute-Guided Remote Sensing Image Detailed Description

This paper proposes an attribute-guided multi-level attention network
(AG-MLAN) to learn fine-grained fashion similarity. AG-MLAN is able to make a
more accurate attribute positioning and capture more discriminative features
under the guidance of the specified attribute. Specifically, the AG-MLAN
contains two branches, branch 1 aims to force the model to recognize different
attributes, while branch 2 aims to learn multiple attribute-specific embedding
spaces for measuring the fine-grained similarity. We first improve the
Convolutional Neural Network (CNN) backbone to extract hierarchical feature
representations, then the extracted feature representations are passed into
branch 1 for attribute classification and branch 2 for multi-level feature
extraction. In branch 2, we first propose a multi-level attention module to
extract a more discriminative representation under the guidance of a specific
attribute. Then, we adopt a masked embedding module to learn attribute-aware
embedding. Finally, the AG-MLAN is trained with a weighted loss of the
classification loss in branch 1 and the triplet loss of the masked embedding
features in branch 2 to further improve the accuracy in attribute location.
Extensive experiments on the DeepFashion, FashionAI, and Zappos50k datasets
show the effectiveness of AG-MLAN for fine-grained fashion similarity learning
and its potential for attribute-guided retrieval tasks. The proposed AG-MLAN
outperforms the state-of-the-art methods in the fine-grained fashion similarity
retrieval task.

本研究提出了一种属性引导的多层注意力网络（AG-MLAN），能够在指定属性的引导下更准确地定位属性位置和捕捉区分性特征，并通过对深度时尚、FashionAI 和 Zappos50k 数据集上的实验验证了其在细粒度时尚相似度学习和属性引导检索任务中的有效性。该模型在细粒度时尚相似度检索任务中的表现超过了现有最优方法。

细粒度时尚检索的属性引导多级注意力网络

Attribute-Guided Multi-Level Attention Network for Fine-Grained Fashion Retrieval

We are interested in attribute-guided face generation: given a low-res face
input image, an attribute vector that can be extracted from a high-res image
(attribute image), our new method generates a high-res face image for the
low-res input that satisfies the given attributes. To address this problem, we
condition the CycleGAN and propose conditional CycleGAN, which is designed to
1) handle unpaired training data because the training low/high-res and high-res
attribute images may not necessarily align with each other, and to 2) allow
easy control of the appearance of the generated face via the input attributes.
We demonstrate impressive results on the attribute-guided conditional CycleGAN,
which can synthesize realistic face images with appearance easily controlled by
user-supplied attributes (e.g., gender, makeup, hair color, eyeglasses). Using
the attribute image as identity to produce the corresponding conditional vector
and by incorporating a face verification network, the attribute-guided network
becomes the identity-guided conditional CycleGAN which produces impressive and
interesting results on identity transfer. We demonstrate three applications on
identity-guided conditional CycleGAN: identity-preserving face superresolution,
face swapping, and frontal face generation, which consistently show the
advantage of our new method.

本文提出了基于属性的条件生成对抗网络（CycleGAN）来生成高分辨率的面部图像，并可以通过输入属性轻松控制所生成的面部外貌。我们还演示了基于该模型的三种应用：身份保留的人脸超分辨率、人脸交换和正面人脸生成，这些应用表明了我们新方法的优势。