Nutrition information plays a pillar role in clinical dietary practice,
precision nutrition, and food industry. Currently, food composition compilation
serves as a standard paradigm to estimate food nutrition information according
to food ingredient information. However, within this paradigm, conventional
approaches are laborious and highly dependent on the experience of data
managers, they cannot keep pace with the dynamic consumer market and resulting
in lagging and missing nutrition data and earlier machine learning methods
unable to fully understand food ingredient statement information or ignored the
characteristic of food image. To this end, we developed a novel vision-language
AI model, UMDFood-VL, using front-of-package labeling and product images to
accurately estimate food composition profiles. In order to drive such large
model training, we established UMDFood-90k, the most comprehensive multimodal
food database to date. The UMDFood-VL model significantly outperformed
convolutional neural networks (CNNs) and recurrent neural networks (RNNs) on a
variety of nutrition value estimations. For instance, we achieved macro-AUCROC
up to 0.921 for fat value estimation, which satisfied the practice requirement
of food composition compilation. This performance shed the light to generalize
to other food and nutrition-related data compilation and catalyzed the
evolution of other food applications.

本文提出了一种基于视觉语言神经网络模型的 UMDFood-VL 模型，并利用最全面的 UMDFood-90k 多模态食品数据库，相较于传统的卷积神经网络（CNNs）和循环神经网络（RNNs），在各种营养价值估计上显著提高，尤其是在食品成分估计上取得了 95％的宏平均 AUCROC，具有一定的实际应用价值。

视觉 - 语言模型推动食品成分编译

Vision-language models boost food composition compilation

Food diary applications represent a tantalizing market. Such applications,
based on image food recognition, opened to new challenges for computer vision
and pattern recognition algorithms. Recent works in the field are focusing
either on hand-crafted representations or on learning these by exploiting deep
neural networks. Despite the success of such a last family of works, these
generally exploit off-the shelf deep architectures to classify food dishes.
Thus, the architectures are not cast to the specific problem. We believe that
better results can be obtained if the deep architecture is defined with respect
to an analysis of the food composition. Following such an intuition, this work
introduces a new deep scheme that is designed to handle the food structure.
Specifically, inspired by the recent success of residual deep network, we
exploit such a learning scheme and introduce a slice convolution block to
capture the vertical food layers. Outputs of the deep residual blocks are
combined with the sliced convolution to produce the classification score for
specific food categories. To evaluate our proposed architecture we have
conducted experimental results on three benchmark datasets. Results demonstrate
that our solution shows better performance with respect to existing approaches
(e.g., a top-1 accuracy of 90.27% on the Food-101 challenging dataset).

本篇论文提出了一种新的深层神经网络结构，通过对食物构成的分析提出了切片卷积块来捕捉垂直的食品层，将深度残差网络的输出与切片卷积相结合来产生特定食品类别的分类得分，并在三个基准数据集上进行实验，展示了与现有方法相比更好的性能。