In this paper, we explore the zero-shot capability of the Segment Anything
Model (SAM) for food image segmentation. To address the lack of class-specific
information in SAM-generated masks, we propose a novel framework, called
FoodSAM. This innovative approach integrates the coarse semantic mask with
SAM-generated masks to enhance semantic segmentation quality. Besides, we
recognize that the ingredients in food can be supposed as independent
individuals, which motivated us to perform instance segmentation on food
images. Furthermore, FoodSAM extends its zero-shot capability to encompass
panoptic segmentation by incorporating an object detector, which renders
FoodSAM to effectively capture non-food object information. Drawing inspiration
from the recent success of promptable segmentation, we also extend FoodSAM to
promptable segmentation, supporting various prompt variants. Consequently,
FoodSAM emerges as an all-encompassing solution capable of segmenting food
items at multiple levels of granularity. Remarkably, this pioneering framework
stands as the first-ever work to achieve instance, panoptic, and promptable
segmentation on food images. Extensive experiments demonstrate the feasibility
and impressing performance of FoodSAM, validating SAM's potential as a
prominent and influential tool within the domain of food image segmentation. We
release our code at this https URL

FoodSAM 是一种创新的框架，整合了粗糙的语义掩膜和 SAM 生成的掩膜，以提高语义分割质量，并将零样本能力扩展到实例分割和全景分割，是首个在食品图像上实现实例、全景和可提示分割的工作。

食品 SAM：任何食品分割

FoodSAM: Any Food Segmentation

Food image segmentation is an important task that has ubiquitous
applications, such as estimating the nutritional value of a plate of food.
Although machine learning models have been used for segmentation in this
domain, food images pose several challenges. One challenge is that food items
can overlap and mix, making them difficult to distinguish. Another challenge is
the degree of inter-class similarity and intra-class variability, which is
caused by the varying preparation methods and dishes a food item may be served
in. Additionally, class imbalance is an inevitable issue in food datasets. To
address these issues, two models are trained and compared, one based on
convolutional neural networks and the other on Bidirectional Encoder
representation for Image Transformers (BEiT). The models are trained and
valuated using the FoodSeg103 dataset, which is identified as a robust
benchmark for food image segmentation. The BEiT model outperforms the previous
state-of-the-art model by achieving a mean intersection over union of 49.4 on
FoodSeg103. This study provides insights into transfering knowledge using
convolution and Transformer-based approaches in the food image domain.

本文研究了食物图像分割的困难，提供了一个鲁棒的基准数据集 FoodSeg103，采用卷积神经网络和双向编码器表示图像转换器（BEiT）进行对比，并证明 BEiT 在食物图像分割中的表现优于其他模型，表明转移学习可以提高图像分割性能。