Large Multi-modal Models (LMMs) have made impressive progress in many
vision-language tasks. Nevertheless, the performance of general LMMs in
specific domains is still far from satisfactory. This paper proposes FoodLMM, a
versatile food assistant based on LMMs with various capabilities, including
food recognition, ingredient recognition, recipe generation, nutrition
estimation, food segmentation and multi-round conversation. To facilitate
FoodLMM to deal with tasks beyond pure text output, we introduce a series of
novel task-specific tokens and heads, enabling the model to predict food
nutritional values and multiple segmentation masks. We adopt a two-stage
training strategy. In the first stage, we utilize multiple public food
benchmarks for multi-task learning by leveraging instruct-following paradigm.
In the second stage, we construct a multi-round conversation and a reasoning
segmentation datasets to fine-tune the model, enabling it to conduct
professional dialogues and generate segmentation masks based on complex
reasoning in food domain. Our fine-tuned FoodLMM achieves state-of-the-art
results across several food benchmarks. We will make our code, models and
datasets publicly available.

提出了一种基于 Large Multi-modal Models 的多功能食品助手 FoodLMM，包括食品识别、成分识别、食谱生成、营养估计、食品分割和多轮对话等各种能力，通过引入一系列新颖的任务特定标记和头部来预测食品营养价值和多个分割掩码，采用两阶段训练策略，使其能够进行专业对话和基于对食品领域的复杂推理来生成分割掩码，并在若干食品数据集上取得了最先进的结果。