Multi-modal large language models (MLLMs) can understand image-language
prompts and demonstrate impressive reasoning ability. In this paper, we extend
MLLMs' output by empowering MLLMs with the segmentation ability. The extended
MLLMs can both output language responses to the image-language prompts and
segment the regions that the complex question or query in the language prompts
focuses on. To this end, the existing work, LISA, enlarges the original word
embeddings with an additional segment token and fine-tunes dialogue generation
and query-focused segmentation together, where the feature of the segment token
is used to prompt the segment-anything model. Although they achieve superior
segmentation performance, we observe that the dialogue ability decreases by a
large margin compared to the original MLLMs. To maintain the original MLLMs'
dialogue ability, we propose a novel MLLMs framework, coined as LLaVASeg, which
leverages a chain-of-thought prompting strategy to instruct the MLLMs to
segment the target region queried by the user. The MLLMs are first prompted to
reason about the simple description of the target region from the complicated
user query, then extract the visual attributes of the target region according
to the understanding of MLLMs to the image. These visual attributes, such as
color and relative locations, are utilized to prompt the downstream
segmentation model. Experiments show that the proposed method keeps the
original dialogue ability and equips the MLLMs' model with strong reasoning
segmentation ability. The code is available at
this https URL

我们扩展了多模态大语言模型（MLLMs）的输出，通过赋予其分割能力，从而使其能够同时输出与图像 - 语言提示相关的语言响应并分割语言提示中复杂问题或查询所关注的区域。我们提出了一种名为 LLaVASeg 的新颖 MLLMs 框架，利用连续思维提示策略指导 MLLMs 对用户查询的目标区域进行分割，从而保持了原始 MLLMs 的对话能力并赋予了 MLLMs 模型强大的推理分割能力。

强化多模态大语言模型的分割能力

Empowering Segmentation Ability to Multi-modal Large Language Models

Inspired by the success of transfer learning in computer vision, roboticists
have investigated visual pre-training as a means to improve the learning
efficiency and generalization ability of policies learned from pixels. To that
end, past work has favored large object interaction datasets, such as
first-person videos of humans completing diverse tasks, in pursuit of
manipulation-relevant features. Although this approach improves the efficiency
of policy learning, it remains unclear how reliable these representations are
in the presence of distribution shifts that arise commonly in robotic
applications. Surprisingly, we find that visual representations designed for
manipulation and control tasks do not necessarily generalize under subtle
changes in lighting and scene texture or the introduction of distractor
objects. To understand what properties do lead to robust representations, we
compare the performance of 15 pre-trained vision models under different visual
appearances. We find that emergent segmentation ability is a strong predictor
of out-of-distribution generalization among ViT models. The rank order induced
by this metric is more predictive than metrics that have previously guided
generalization research within computer vision and machine learning, such as
downstream ImageNet accuracy, in-domain accuracy, or shape-bias as evaluated by
cue-conflict performance. We test this finding extensively on a suite of
distribution shifts in ten tasks across two simulated manipulation
environments. On the ALOHA setup, segmentation score predicts real-world
performance after offline training with 50 demonstrations.

通过对 15 个预训练视觉模型的性能比较，发现视觉出现分割能力是 ViT 模型在分布偏移下的强预测因子。在十个任务中进行广泛测试后，分割分数在离线训练和 50 次演示后预测了真实世界的性能。

预训练视觉表示对稳健操控的成功因素

What Makes Pre-Trained Visual Representations Successful for Robust  Manipulation?

In high-quality radiotherapy delivery, precise segmentation of targets and
healthy structures is essential. This study proposes Radiomics features as a
superior measure for assessing the segmentation ability of physicians and
auto-segmentation tools, in comparison to the widely used Dice Similarity
Coefficient (DSC). The research involves selecting reproducible radiomics
features for evaluating segmentation accuracy by analyzing radiomics data from
2 CT scans of 10 lung tumors, available in the RIDER Data Library. Radiomics
features were extracted using PyRadiomics, with selection based on the
Concordance Correlation Coefficient (CCC). Subsequently, CT images from 10
patients, each segmented by different physicians or auto-segmentation tools,
were used to assess segmentation performance. The study reveals 206 radiomics
features with a CCC greater than 0.93 between the two CT images, indicating
robust reproducibility. Among these features, seven exhibit low Intraclass
Correlation Coefficients (ICC), signifying increased sensitivity to
segmentation differences. Notably, ICCs of original shape features, including
sphericity, elongation, and flatness, ranged from 0.1177 to 0.995. In contrast,
all DSC values exceeded 0.778. This research demonstrates that radiomics
features, particularly those related to shape and energy, can capture subtle
variations in tumor segmentation characteristics, unlike DSC. As a result,
Radiomics features with ICC prove superior for evaluating a physician's tumor
segmentation ability and the performance of auto-segmentation tools. The
findings suggest that these new metrics can be employed to assess novel
auto-segmentation methods and enhance the training of individuals in medical
segmentation, thus contributing to improved radiotherapy practices.

该研究使用放射组学特征作为评估医生和自动分割工具的分割能力的最优指标，与 Dice 相似系数（DSC）相比较。研究选择可再现的放射组学特征来评估分割准确性，并使用 CT 扫描数据从 10 个肺肿瘤中提取的放射组学特征进行分析。研究发现，放射组学特征能够捕捉肿瘤分割特征的细微变化，尤其与形状和能量相关的特征表现出较高的敏感度，因此可用于评估医生的分割能力和自动分割工具的性能。这些发现可用于评估新的自动分割方法，提高医学分割培训水平，从而改进放射治疗实践。