Generalized Referring Expression Segmentation (GRES) extends the scope of
classic RES to referring to multiple objects in one expression or identifying
the empty targets absent in the image. GRES poses challenges in modeling the
complex spatial relationships of the instances in the image and identifying
non-existing referents. Recently, Multimodal Large Language Models (MLLMs) have
shown tremendous progress in these complicated vision-language tasks.
Connecting Large Language Models (LLMs) and vision models, MLLMs are proficient
in understanding contexts with visual inputs. Among them, LISA, as a
representative, adopts a special [SEG] token to prompt a segmentation mask
decoder, e.g., SAM, to enable MLLMs in the RES task. However, existing
solutions to of GRES remain unsatisfactory since current segmentation MLLMs
cannot properly handle the cases where users might reference multiple subjects
in a singular prompt or provide descriptions incongruent with any image target.
In this paper, we propose Generalized Segmentation Vision Assistant (GSVA) to
address this gap. Specifically, GSVA reuses the [SEG] token to prompt the
segmentation model towards supporting multiple mask references simultaneously
and innovatively learns to generate a [REJ] token to reject the null targets
explicitly. Experiments validate GSVA's efficacy in resolving the GRES issue,
marking a notable enhancement and setting a new record on the GRES benchmark
gRefCOCO dataset. GSVA also proves effective across various classic referring
expression segmentation and comprehension tasks.

该研究提出了广义引用表达分割（GRES）的问题，并引入了 Multimodal Large Language Models（MLLMs）的方法来解决这个问题，最终通过 Generalized Segmentation Vision Assistant（GSVA）在 gRefCOCO 数据集上取得了显著的改进和新的记录。

GSVA：通过多模态大型语言模型进行广义分割

GSVA: Generalized Segmentation via Multimodal Large Language Models

Although perception systems have made remarkable advancements in recent
years, they still rely on explicit human instruction to identify the target
objects or categories before executing visual recognition tasks. Such systems
lack the ability to actively reason and comprehend implicit user intentions. In
this work, we propose a new segmentation task -- reasoning segmentation. The
task is designed to output a segmentation mask given a complex and implicit
query text. Furthermore, we establish a benchmark comprising over one thousand
image-instruction pairs, incorporating intricate reasoning and world knowledge
for evaluation purposes. Finally, we present LISA: large Language Instructed
Segmentation Assistant, which inherits the language generation capabilities of
the multi-modal Large Language Model (LLM) while also possessing the ability to
produce segmentation masks. We expand the original vocabulary with a <SEG>
token and propose the embedding-as-mask paradigm to unlock the segmentation
capability. Remarkably, LISA can handle cases involving: 1) complex reasoning;
2) world knowledge; 3) explanatory answers; 4) multi-turn conversation. Also,
it demonstrates robust zero-shot capability when trained exclusively on
reasoning-free datasets. In addition, fine-tuning the model with merely 239
reasoning segmentation image-instruction pairs results in further performance
enhancement. Experiments show our method not only unlocks new reasoning
segmentation capabilities but also proves effective in both complex reasoning
segmentation and standard referring segmentation tasks. Code, models, and demo
are at this https URL

该研究提出了一种新的推理分割任务，以激活感知系统中的推理分割能力，并展示了多模态语言模型 LISA 在复杂推理分割和标准引用分割任务中的有效性。

LISA：利用大型语言模型进行分割推理

LISA: Reasoning Segmentation via Large Language Model

Out-of-distribution (OOD) generalization is an important issue for Graph
Neural Networks (GNNs). Recent works employ different graph editions to
generate augmented environments and learn an invariant GNN for generalization.
However, the label shift usually occurs in augmentation since graph structural
edition inevitably alters the graph label. This brings inconsistent predictive
relationships among augmented environments, which is harmful to generalization.
To address this issue, we propose \textbf{LiSA}, which generates
label-invariant augmentations to facilitate graph OOD generalization. Instead
of resorting to graph editions, LiSA exploits \textbf{L}abel-\textbf{i}nvariant
\textbf{S}ubgraphs of the training graphs to construct \textbf{A}ugmented
environments. Specifically, LiSA first designs the variational subgraph
generators to extract locally predictive patterns and construct multiple
label-invariant subgraphs efficiently. Then, the subgraphs produced by
different generators are collected to build different augmented environments.
To promote diversity among augmented environments, LiSA further introduces a
tractable energy-based regularization to enlarge pair-wise distances between
the distributions of environments. In this manner, LiSA generates diverse
augmented environments with a consistent predictive relationship and
facilitates learning an invariant GNN. Extensive experiments on node-level and
graph-level OOD benchmarks show that LiSA achieves impressive generalization
performance with different GNN backbones. Code is available on
https://github.com/Samyu0304/LiSA.

LiSA 通过生成标签不变增强的方式解决了图神经网络中图结构编辑带来的标签转换问题，实现了一致且多样化的预测关系来方便图结构的 OOD 泛化。

基于数据增强的图形 OOD 泛化中注意标签偏移

Mind the Label Shift of Augmentation-based Graph OOD Generalization

Machine learning algorithms typically assume that training and test examples
are drawn from the same distribution. However, distribution shift is a common
problem in real-world applications and can cause models to perform dramatically
worse at test time. In this paper, we specifically consider the problems of
subpopulation shifts (e.g., imbalanced data) and domain shifts. While prior
works often seek to explicitly regularize internal representations or
predictors of the model to be domain invariant, we instead aim to learn
invariant predictors without restricting the model's internal representations
or predictors. This leads to a simple mixup-based technique which learns
invariant predictors via selective augmentation called LISA. LISA selectively
interpolates samples either with the same labels but different domains or with
the same domain but different labels. Empirically, we study the effectiveness
of LISA on nine benchmarks ranging from subpopulation shifts to domain shifts,
and we find that LISA consistently outperforms other state-of-the-art methods
and leads to more invariant predictors. We further analyze a linear setting and
theoretically show how LISA leads to a smaller worst-group error.

探讨了在实际应用中常见的基于配分转移的问题，提出了一种选择性增广的简单混合技术 LISA，通过学习不受限制的内部表示或预测器来学习不变的预测器，并在包括亚种人群转移和领域转移的九个基准测试中验证了 LISA 的有效性。