Given the increasing complexity of AI applications, traditional spatial
architectures frequently fall short. Our analysis identifies a pattern of
interconnected, multi-faceted tasks encompassing both AI and general
computational processes. In response, we have conceptualized "Orchestrated AI
Workflows," an approach that integrates various tasks with logic-driven
decisions into dynamic, sophisticated workflows. Specifically, we find that the
intrinsic Dual Dynamicity of Orchestrated AI Workflows, namely dynamic
execution times and frequencies of Task Blocks, can be effectively represented
using the Orchestrated Workflow Graph. Furthermore, the intrinsic Dual
Dynamicity poses challenges to existing spatial architecture, namely
Indiscriminate Resource Allocation, Reactive Load Rebalancing, and Contagious
PEA Idleness.
To overcome these challenges, we present Octopus, a scale-out spatial
architecture and a suite of advanced scheduling strategies optimized for
executing Orchestrated AI Workflows, such as the Discriminate Dual-Scheduling
Mechanism, Adaptive TBU Scheduling Strategy, and Proactive Cluster Scheduling
Strategy. Our evaluations demonstrate that Octopus significantly outperforms
traditional architectures in handling the dynamic demands of Orchestrated AI
Workflows, and possesses robust scalability in large scale hardware such as
wafer-scale chip.

给定了日益复杂的人工智能应用，我们通过分析发现了一种相互连接的、多方面任务的模式，涵盖了人工智能和一般计算过程。对此，我们提出了 “Orchestrated AI Workflows” 的概念，将各种任务与逻辑驱动的决策整合成动态而复杂的工作流程。同时，我们提出了 Octopus，一个针对执行 Orchestrated AI Workflows 而优化的可扩展的空间架构和一套先进的调度策略，它在处理动态需求方面显著优于传统架构，并在大规模硬件（如大规模芯片）上具有强大的可扩展性。

规模化分布式空间架构上高效 AI 工作流执行

Efficient Orchestrated AI Workflows Execution on Scale-out Spatial  Architecture

Large vision-language models (VLMs) have achieved substantial progress in
multimodal perception and reasoning. Furthermore, when seamlessly integrated
into an embodied agent, it signifies a crucial stride towards the creation of
autonomous and context-aware systems capable of formulating plans and executing
commands with precision. In this paper, we introduce Octopus, a novel VLM
designed to proficiently decipher an agent's vision and textual task objectives
and to formulate intricate action sequences and generate executable code. Our
design allows the agent to adeptly handle a wide spectrum of tasks, ranging
from mundane daily chores in simulators to sophisticated interactions in
complex video games. Octopus is trained by leveraging GPT-4 to control an
explorative agent to generate training data, i.e., action blueprints and the
corresponding executable code, within our experimental environment called
OctoVerse. We also collect the feedback that allows the enhanced training
scheme of Reinforcement Learning with Environmental Feedback (RLEF). Through a
series of experiments, we illuminate Octopus's functionality and present
compelling results, and the proposed RLEF turns out to refine the agent's
decision-making. By open-sourcing our model architecture, simulator, and
dataset, we aspire to ignite further innovation and foster collaborative
applications within the broader embodied AI community.

Octopus 是一种新颖的大视觉 - 语言模型，能够有效地解读代理的视觉和文本任务目标，并制定复杂的行动序列和生成可执行代码，通过开源模型结构、模拟器和数据集，旨在激发更多创新，并在更广泛的具身化人工智能社区中促进协作应用。

章鱼：基于环境反馈的视觉语言程序员

Octopus: Embodied Vision-Language Programmer from Environmental Feedback

Efficient vision works maximize accuracy under a latency budget. These works
evaluate accuracy offline, one image at a time. However, real-time vision
applications like autonomous driving operate in streaming settings, where
ground truth changes between inference start and finish. This results in a
significant accuracy drop. Therefore, a recent work proposed to maximize
accuracy in streaming settings on average. In this paper, we propose to
maximize streaming accuracy for every environment context. We posit that
scenario difficulty influences the initial (offline) accuracy difference, while
obstacle displacement in the scene affects the subsequent accuracy degradation.
Our method, Octopus, uses these scenario properties to select configurations
that maximize streaming accuracy at test time. Our method improves tracking
performance (S-MOTA) by 7.4% over the conventional static approach. Further,
performance improvement using our method comes in addition to, and not instead
of, advances in offline accuracy.

本文提出了一种新方法，Octopus，根据场景难度和障碍物位移等因素来选择配置，以最大化流媒体环境下的跟踪精度 (S-MOTA)，进一步提高离线准确性的同时，将跟踪性能提高了 7.4％。