Informational videos serve as a crucial source for explaining conceptual and
procedural knowledge to novices and experts alike. When producing informational
videos, editors edit videos by overlaying text/images or trimming footage to
enhance the video quality and make it more engaging. However, video editing can
be difficult and time-consuming, especially for novice video editors who often
struggle with expressing and implementing their editing ideas. To address this
challenge, we first explored how multimodality$-$natural language (NL) and
sketching, which are natural modalities humans use for expression$-$can be
utilized to support video editors in expressing video editing ideas. We
gathered 176 multimodal expressions of editing commands from 10 video editors,
which revealed the patterns of use of NL and sketching in describing edit
intents. Based on the findings, we present ExpressEdit, a system that enables
editing videos via NL text and sketching on the video frame. Powered by LLM and
vision models, the system interprets (1) temporal, (2) spatial, and (3)
operational references in an NL command and spatial references from sketching.
The system implements the interpreted edits, which then the user can iterate
on. An observational study (N=10) showed that ExpressEdit enhanced the ability
of novice video editors to express and implement their edit ideas. The system
allowed participants to perform edits more efficiently and generate more ideas
by generating edits based on user's multimodal edit commands and supporting
iterations on the editing commands. This work offers insights into the design
of future multimodal interfaces and AI-based pipelines for video editing.

通过自然语言和草图等多模式表达，提出了一种支持视频编辑的系统 ExpressEdit，该系统通过解释自然语言命令和草图实现解释性编辑，增强了初学者视频编辑者表达和实现编辑理念的能力，为未来多模态界面和基于人工智能的视频编辑管道的设计提供了洞见。

ExpressEdit：自然语言和示意草图的视频编辑

ExpressEdit: Video Editing with Natural Language and Sketching

Supporting voice commands in applications presents significant benefits to
users. However, adding such support to existing GUI-based web apps is
effort-consuming with a high learning barrier, as shown in our formative study,
due to the lack of unified support for creating multimodal interfaces. We
present Geno---a developer tool for adding the voice input modality to existing
web apps without requiring significant NLP expertise. Geno provides a
high-level workflow for developers to specify functionalities to be supported
by voice (intents), create language models for detecting intents and the
relevant information (parameters) from user utterances, and fulfill the intents
by either programmatically invoking the corresponding functions or replaying
GUI actions on the web app. Geno further supports multimodal references to GUI
context in voice commands (e.g. "move this [event] to next week" while pointing
at an event with the cursor). In a study, developers with little NLP expertise
were able to add multimodal voice command support for two existing web apps
using Geno.

本文介绍了 Geno 开发工具，该工具可以帮助开发人员在不需要过多自然语言处理专业知识的情况下，将语音输入功能添加到现有的基于 GUI 的 web 应用程序中，支持多模态引用 GUI 上下文的语音命令。