A rapidly growing amount of content posted online, such as food recipes,
opens doors to new exciting applications at the intersection of vision and
language. In this work, we aim to estimate the calorie amount of a meal
directly from an image by learning from recipes people have published on the
Internet, thus skipping time-consuming manual data annotation. Since there are
few large-scale publicly available datasets captured in unconstrained
environments, we propose the pic2kcal benchmark comprising 308,000 images from
over 70,000 recipes including photographs, ingredients and instructions. To
obtain nutritional information of the ingredients and automatically determine
the ground-truth calorie value, we match the items in the recipes with
structured information from a food item database.
We evaluate various neural networks for regression of the calorie quantity
and extend them with the multi-task paradigm. Our learning procedure combines
the calorie estimation with prediction of proteins, carbohydrates, and fat
amounts as well as a multi-label ingredient classification. Our experiments
demonstrate clear benefits of multi-task learning for calorie estimation,
surpassing the single-task calorie regression by 9.9%. To encourage further
research on this task, we make the code for generating the dataset and the
models publicly available.

本研究提出 pic2kcal 基准，通过学习人们在互联网上发布的食谱来直接从图像中估算餐的卡路里含量，并通过多任务范式扩展神经网络进行蛋白质、碳水化合物和脂肪量预测以及多标签成分分类。实验表明，多任务学习可以显著提高卡路里估计，优于单任务卡路里回归 9.9%。