Conventional approaches to dietary assessment are primarily grounded in
self-reporting methods or structured interviews conducted under the supervision
of dietitians. These methods, however, are often subjective, potentially
inaccurate, and time-intensive. Although artificial intelligence (AI)-based
solutions have been devised to automate the dietary assessment process, these
prior AI methodologies encounter challenges in their ability to generalize
across a diverse range of food types, dietary behaviors, and cultural contexts.
This results in AI applications in the dietary field that possess a narrow
specialization and limited accuracy. Recently, the emergence of multimodal
foundation models such as GPT-4V powering the latest ChatGPT has exhibited
transformative potential across a wide range of tasks (e.g., Scene
understanding and image captioning) in numerous research domains. These models
have demonstrated remarkable generalist intelligence and accuracy, capable of
processing various data modalities. In this study, we explore the application
of multimodal ChatGPT within the realm of dietary assessment. Our findings
reveal that GPT-4V excels in food detection under challenging conditions with
accuracy up to 87.5% without any fine-tuning or adaptation using food-specific
datasets. By guiding the model with specific language prompts (e.g., African
cuisine), it shifts from recognizing common staples like rice and bread to
accurately identifying regional dishes like banku and ugali. Another GPT-4V's
standout feature is its contextual awareness. GPT-4V can leverage surrounding
objects as scale references to deduce the portion sizes of food items, further
enhancing its accuracy in translating food weight into nutritional content.
This alignment with the USDA National Nutrient Database underscores GPT-4V's
potential to advance nutritional science and dietary assessment techniques.

常规的饮食评估方法主要基于自述法或在饮食师监督下进行的结构化面谈，然而这些方法往往主观、有可能不准确且耗时，最近的多模态 ChatGPT 研究表明其在特定语境下的食物检测准确性高达 87.5%，并能通过周围物体作为参考来推断食物的份量，进一步提高了翻译食物重量为营养内容的准确性。

多模式 ChatGPT 进行饮食评估的系统分析

Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis

Understanding the nutritional content of food from visual data is a
challenging computer vision problem, with the potential to have a positive and
widespread impact on public health. Studies in this area are limited to
existing datasets in the field that lack sufficient diversity or labels
required for training models with nutritional understanding capability. We
introduce Nutrition5k, a novel dataset of 5k diverse, real world food dishes
with corresponding video streams, depth images, component weights, and high
accuracy nutritional content annotation. We demonstrate the potential of this
dataset by training a computer vision algorithm capable of predicting the
caloric and macronutrient values of a complex, real world dish at an accuracy
that outperforms professional nutritionists. Further we present a baseline for
incorporating depth sensor data to improve nutrition predictions. We will
publicly release Nutrition5k in the hope that it will accelerate innovation in
the space of nutritional understanding.

该研究通过引入 Nutrition5k 数据集，采用计算机视觉算法能够精确预测真实世界中的复杂菜肴的热量和营养素值，并且具有超过专业营养学家的准确性，使这一领域的营养理解得到加速创新。