Long-form video understanding represents a significant challenge within
computer vision, demanding a model capable of reasoning over long multi-modal
sequences. Motivated by the human cognitive process for long-form video
understanding, we emphasize interactive reasoning and planning over the ability
to process lengthy visual inputs. We introduce a novel agent-based system,
VideoAgent, that employs a large language model as a central agent to
iteratively identify and compile crucial information to answer a question, with
vision-language foundation models serving as tools to translate and retrieve
visual information. Evaluated on the challenging EgoSchema and NExT-QA
benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only
8.4 and 8.2 frames used on average. These results demonstrate superior
effectiveness and efficiency of our method over the current state-of-the-art
methods, highlighting the potential of agent-based approaches in advancing
long-form video understanding.

利用一个新型的基于代理的系统，以长篇视频理解为挑战，通过交互性推理和规划来处理长时间的多模式序列，同时使用大型语言模型作为中央代理来识别和编译关键信息以回答问题，视觉语言基础模型用于翻译和检索视觉信息。在具有挑战性的 EgoSchema 和 NExT-QA 基准测试中，VideoAgent 的零样本准确率分别达到了 54.1% 和 71.3%，仅使用平均 8.4 和 8.2 帧。这些结果证明了我们方法在效果和效率上优于当前技术水平，突显了基于代理的方法在提升长篇视频理解方面的潜力。

VideoAgent: 基于大型语言模型的长视频理解

VideoAgent: Long-form Video Understanding with Large Language Model as  Agent

A standard ML model is commonly generated by a single method that specifies
aspects such as architecture, initialization, training data and hyperparameters
configuration. The presented work introduces a novel methodology allowing to
define multiple methods as distinct agents. Agents can collaborate and compete
to generate and improve ML models for a given tasks. The proposed methodology
is demonstrated with the generation and extension of a dynamic modular
multitask ML system solving more than one hundred image classification tasks.
Diverse agents can compete to produce the best performing model for a task by
reusing the modules introduced to the system by competing agents. The presented
work focuses on the study of agents capable of: 1) reusing the modules
generated by concurrent agents, 2) activating in parallel multiple modules in a
frozen state by connecting them with trainable modules, 3) condition the
activation mixture on each data sample by using a trainable router module. We
demonstrate that this simple per-sample parallel routing method can boost the
quality of the combined solutions by training a fraction of the activated
parameters.

该研究介绍了一种新的机器学习模型生成方法，基于 agent-based system 的思想，可以通过并行计算和合作竞争的方式为多个图像分类任务生成和扩展多模块的深度学习模型。在针对每个数据样本进行的简单并行路由方法的训练下，可以提高组合解决方案的质量。

用于模块化多任务机器学习系统的多路径代理

Multipath agents for modular multitask ML systems

We apply the knowledge of urban settings established with the study of Land
Use and Transport Interaction (LUTI) models to develop reward functions for an
agent-based system capable of planning realistic artificial cities. The system
aims to replicate in the micro scale the main components of real settlements,
such as zoning and accessibility in a road network. Moreover, we propose a
novel representation for the agent's environment that efficiently combines the
road graph with a discrete model for the land. Our system starts from an empty
map consisting only of the road network graph, and the agent incrementally
expands it by building new sites while distinguishing land uses between
residential, commercial, industrial, and recreational.

利用 LUTI 模型中对城市环境的知识为基础，结合代理系统，设计了能够规划逼真的人工城市的奖励函数，实现了对真实城镇环境中的分区和道路可达性等主要组成部分的微观级别复制；我们提出了代理环境的一种新颖表示形式，将道路图与土地的离散模型高效地结合起来，并通过分辨居住用地、商业用地、工业用地和娱乐用地，从空白图开始逐渐扩展地图。