The proliferation of mobile devices and social media has revolutionized
content dissemination, with short-form video becoming increasingly prevalent.
This shift has introduced the challenge of video reframing to fit various
screen aspect ratios, a process that highlights the most compelling parts of a
video. Traditionally, video reframing is a manual, time-consuming task
requiring professional expertise, which incurs high production costs. A
potential solution is to adopt some machine learning models, such as video
salient object detection, to automate the process. However, these methods often
lack generalizability due to their reliance on specific training data. The
advent of powerful large language models (LLMs) open new avenues for AI
capabilities. Building on this, we introduce Reframe Any Video Agent (RAVA), a
LLM-based agent that leverages visual foundation models and human instructions
to restructure visual content for video reframing. RAVA operates in three
stages: perception, where it interprets user instructions and video content;
planning, where it determines aspect ratios and reframing strategies; and
execution, where it invokes the editing tools to produce the final video. Our
experiments validate the effectiveness of RAVA in video salient object
detection and real-world reframing tasks, demonstrating its potential as a tool
for AI-powered video editing.

通过使用大型语言模型 LLM，本研究提出了一种基于视觉基础模型和人类指导的 Reframe Any Video Agent (RAVA) 来自动化视频重构过程，实验证明了 RAVA 在视频显著对象检测和实际重构任务上的有效性，展示了其作为 AI-powered 视频编辑工具的潜力。