Large language models (LLMs) and large visual language models (LVLMs) have
been at the forefront of the artificial intelligence field, particularly for
tasks like text generation, video captioning, and question-answering.
Typically, it is more applicable to train these models on broader knowledge
bases or datasets to increase generalizability, learn relationships between
topics, and recognize patterns. Instead, we propose to provide instructional
datasets specific to the task of each modality within a distinct domain and
then fine-tune the parameters of the model using LORA. With our approach, we
can eliminate all noise irrelevant to the given task while also ensuring that
the model generates with enhanced precision. For this work, we use Video-LLaVA
to generate recipes given cooking videos without transcripts. Video-LLaVA's
multimodal architecture allows us to provide cooking images to its image
encoder, cooking videos to its video encoder, and general cooking questions to
its text encoder. Thus, we aim to remove all noise unrelated to cooking while
improving our model's capabilities to generate specific ingredient lists and
detailed instructions. As a result, our approach to fine-tuning Video-LLaVA
leads to gains over the baseline Video-LLaVA by 2% on the YouCook2 dataset.
While this may seem like a marginal increase, our model trains on an image
instruction dataset 2.5% the size of Video-LLaVA's and a video instruction
dataset 23.76% of Video-LLaVA's.

通过使用 LORA 方法，我们提出了使用特定领域的信息数据集进行模型参数微调的方法，我们的研究旨在提高 Video-LLaVA 模型在烹饪视频中生成特定食材列表和详细指导的能力。