Large language models (LLMs) and large visual language models (LVLMs) have
been at the forefront of the artificial intelligence field, particularly for
tasks like text generation, video captioning, and question-answering.
Typically, it is more applicable to train these models on broader knowledge
bases or datasets to increase generalizability, learn relationships between
topics, and recognize patterns. Instead, we propose to provide instructional
datasets specific to the task of each modality within a distinct domain and
then fine-tune the parameters of the model using LORA. With our approach, we
can eliminate all noise irrelevant to the given task while also ensuring that
the model generates with enhanced precision. For this work, we use Video-LLaVA
to generate recipes given cooking videos without transcripts. Video-LLaVA's
multimodal architecture allows us to provide cooking images to its image
encoder, cooking videos to its video encoder, and general cooking questions to
its text encoder. Thus, we aim to remove all noise unrelated to cooking while
improving our model's capabilities to generate specific ingredient lists and
detailed instructions. As a result, our approach to fine-tuning Video-LLaVA
leads to gains over the baseline Video-LLaVA by 2% on the YouCook2 dataset.
While this may seem like a marginal increase, our model trains on an image
instruction dataset 2.5% the size of Video-LLaVA's and a video instruction
dataset 23.76% of Video-LLaVA's.

通过使用 LORA 方法，我们提出了使用特定领域的信息数据集进行模型参数微调的方法，我们的研究旨在提高 Video-LLaVA 模型在烹饪视频中生成特定食材列表和详细指导的能力。

定向领域微调：为特定训练任务定制分开的模态

Directed Domain Fine-Tuning: Tailoring Separate Modalities for Specific  Training Tasks

This paper tackles recipe generation from unsegmented cooking videos, a task
that requires agents to (1) extract key events in completing the dish and (2)
generate sentences for the extracted events. Our task is similar to dense video
captioning (DVC), which aims at detecting events thoroughly and generating
sentences for them. However, unlike DVC, in recipe generation, recipe story
awareness is crucial, and a model should output an appropriate number of key
events in the correct order. We analyze the output of the DVC model and observe
that although (1) several events are adoptable as a recipe story, (2) the
generated sentences for such events are not grounded in the visual content.
Based on this, we hypothesize that we can obtain correct recipes by selecting
oracle events from the output events of the DVC model and re-generating
sentences for them. To achieve this, we propose a novel transformer-based joint
approach of training event selector and sentence generator for selecting oracle
events from the outputs of the DVC model and generating grounded sentences for
the events, respectively. In addition, we extend the model by including
ingredients to generate more accurate recipes. The experimental results show
that the proposed method outperforms state-of-the-art DVC models. We also
confirm that, by modeling the recipe in a story-aware manner, the proposed
model output the appropriate number of events in the correct order.

本研究提出一种基于转换器的联合方法来生成食谱，该方法可以从未经分段的烹饪视频中提取关键事件并为其生成相关的句子，同时加入食材信息以提高准确性。实验证明该方法优于现有的传统方法且能够正确输出相关事件和句子。

从未分割的烹饪视频中生成食谱

Recipe Generation from Unsegmented Cooking Videos

Watching instructional videos are often used to learn about procedures. Video
captioning is one way of automatically collecting such knowledge. However, it
provides only an indirect, overall evaluation of multimodal models with no
finer-grained quantitative measure of what they have learned. We propose
instead, a benchmark of structured procedural knowledge extracted from cooking
videos. This work is complementary to existing tasks, but requires models to
produce interpretable structured knowledge in the form of verb-argument tuples.
Our manually annotated open-vocabulary resource includes 356 instructional
cooking videos and 15,523 video clip/sentence-level annotations. Our analysis
shows that the proposed task is challenging and standard modeling approaches
like unsupervised segmentation, semantic role labeling, and visual action
detection perform poorly when forced to predict every action of a procedure in
a structured form.

本文提出了从烹饪视频中提取结构化过程知识的基准测试，研究了现有模式的性能。