Conventional approaches to dietary assessment are primarily grounded in
self-reporting methods or structured interviews conducted under the supervision
of dietitians. These methods, however, are often subjective, potentially
inaccurate, and time-intensive. Although artificial intelligence (AI)-based
solutions have been devised to automate the dietary assessment process, these
prior AI methodologies encounter challenges in their ability to generalize
across a diverse range of food types, dietary behaviors, and cultural contexts.
This results in AI applications in the dietary field that possess a narrow
specialization and limited accuracy. Recently, the emergence of multimodal
foundation models such as GPT-4V powering the latest ChatGPT has exhibited
transformative potential across a wide range of tasks (e.g., Scene
understanding and image captioning) in numerous research domains. These models
have demonstrated remarkable generalist intelligence and accuracy, capable of
processing various data modalities. In this study, we explore the application
of multimodal ChatGPT within the realm of dietary assessment. Our findings
reveal that GPT-4V excels in food detection under challenging conditions with
accuracy up to 87.5% without any fine-tuning or adaptation using food-specific
datasets. By guiding the model with specific language prompts (e.g., African
cuisine), it shifts from recognizing common staples like rice and bread to
accurately identifying regional dishes like banku and ugali. Another GPT-4V's
standout feature is its contextual awareness. GPT-4V can leverage surrounding
objects as scale references to deduce the portion sizes of food items, further
enhancing its accuracy in translating food weight into nutritional content.
This alignment with the USDA National Nutrient Database underscores GPT-4V's
potential to advance nutritional science and dietary assessment techniques.

常规的饮食评估方法主要基于自述法或在饮食师监督下进行的结构化面谈，然而这些方法往往主观、有可能不准确且耗时，最近的多模态 ChatGPT 研究表明其在特定语境下的食物检测准确性高达 87.5%，并能通过周围物体作为参考来推断食物的份量，进一步提高了翻译食物重量为营养内容的准确性。

多模式 ChatGPT 进行饮食评估的系统分析

Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis

Food detection is becoming a fundamental task in food computing that supports
various multimedia applications, including food recommendation and dietary
monitoring. To deal with real-world scenarios, food detection needs to localize
and recognize novel food objects that are not seen during training, demanding
Zero-Shot Detection (ZSD). However, the complexity of semantic attributes and
intra-class feature diversity poses challenges for ZSD methods in
distinguishing fine-grained food classes. To tackle this, we propose the
Semantic Separable Diffusion Synthesizer (SeeDS) framework for Zero-Shot Food
Detection (ZSFD). SeeDS consists of two modules: a Semantic Separable
Synthesizing Module (S$^3$M) and a Region Feature Denoising Diffusion Model
(RFDDM). The S$^3$M learns the disentangled semantic representation for complex
food attributes from ingredients and cuisines, and synthesizes discriminative
food features via enhanced semantic information. The RFDDM utilizes a novel
diffusion model to generate diversified region features and enhances ZSFD via
fine-grained synthesized features. Extensive experiments show the
state-of-the-art ZSFD performance of our proposed method on two food datasets,
ZSFooD and UECFOOD-256. Moreover, SeeDS also maintains effectiveness on general
ZSD datasets, PASCAL VOC and MS COCO. The code and dataset can be found at
this https URL

提出了一种基于语义可分离扩散合成器框架的零样本食物检测方法，其通过学习复杂食物属性的语义表征并通过增强的语义信息合成辨别性食物特征。通过生成多样化的区域特征，通过细粒度的综合特征提高零样本食物检测性能。在两个食物数据集上展示了该方法在零样本食物检测上的最新性能。