Different fields in applied machine learning such as computer vision, speech
or natural language processing have been building domain-specialised solutions.
Currently, we are witnessing an opposing trend towards developing more
generalist architectures, driven by Large Language Models and multi-modal
foundational models. These architectures are designed to tackle a variety of
tasks, including those previously unseen and using inputs across multiple
modalities. Taking this trend of generalization to the extreme suggests the
possibility of a single deep network architecture capable of solving all tasks.
This position paper aims to explore developing such a unified architecture and
proposes a theoretical framework of how it could be constructed. Our proposal
is based on the following assumptions. Firstly, tasks are solved by following a
sequence of instructions, typically implemented in code for conventional
computing hardware, which inherently operates sequentially. Second, recent
Generative AI, especially Transformer-based models, demonstrate potential as an
architecture capable of constructing algorithms for a wide range of domains.
For example, GPT-4 shows exceptional capability at in-context learning of novel
tasks which is hard to explain in any other way than the ability to compose
novel solutions from fragments on previously learnt algorithms. Third, the
observation that the main missing component in developing a truly generalised
network is an efficient approach for self-consistent input of previously learnt
sub-steps of an algorithm and their (implicit) composition during the network's
internal forward pass. Our exploration delves into current capabilities and
limitations of Transformer-based and other methods in efficient and correct
algorithm composition and proposes a Transformer-like architecture as well as a
discrete learning framework to overcome these limitations.

提出一种理论框架构建统一的深度网络架构，基于 Transformer 模型以及离散学习框架，以解决当前构建通用网络的挑战。

通过隐含组合进行算法归纳的任务无关架构

Task Agnostic Architecture for Algorithm Induction via Implicit  Composition

Recently, a few open-vocabulary methods have been proposed by employing a
unified architecture to tackle generic segmentation and detection tasks.
However, their performance still lags behind the task-specific models due to
the conflict between different tasks, and their open-vocabulary capability is
limited due to the inadequate use of CLIP. To address these challenges, we
present a universal transformer-based framework, abbreviated as OpenSD, which
utilizes the same architecture and network parameters to handle open-vocabulary
segmentation and detection tasks. First, we introduce a decoder decoupled
learning strategy to alleviate the semantic conflict between thing and staff
categories so that each individual task can be learned more effectively under
the same framework. Second, to better leverage CLIP for end-to-end segmentation
and detection, we propose dual classifiers to handle the in-vocabulary domain
and out-of-vocabulary domain, respectively. The text encoder is further trained
to be region-aware for both thing and stuff categories through decoupled prompt
learning, enabling them to filter out duplicated and low-quality predictions,
which is important to end-to-end segmentation and detection. Extensive
experiments are conducted on multiple datasets under various circumstances. The
results demonstrate that OpenSD outperforms state-of-the-art open-vocabulary
segmentation and detection methods in both closed- and open-vocabulary
settings. Code is available at this https URL

我们提出了一个名为 OpenSD 的通用 Transformer 基础框架，利用相同的架构和网络参数来处理开放词汇分割和检测任务。OpenSD 在封闭和开放词汇设置下，优于最先进的开放词汇分割和检测方法。

OpenSD: 统一的开放词汇分割和检测

OpenSD: Unified Open-Vocabulary Segmentation and Detection

Art forms such as movies and television (TV) dramas are reflections of the
real world, which have attracted much attention from the multimodal learning
community recently. However, existing corpora in this domain share three
limitations: (1) annotated in a scene-oriented fashion, they ignore the
coherence within plots; (2) their text lacks empathy and seldom mentions
situational context; (3) their video clips fail to cover long-form relationship
due to short duration. To address these fundamental issues, using 1,106 TV
drama episodes and 24,875 informative plot-focused sentences written by
professionals, with the help of 449 human annotators, we constructed PTVD, the
first plot-oriented multimodal dataset in the TV domain. It is also the first
non-English dataset of its kind. Additionally, PTVD contains more than 26
million bullet screen comments (BSCs), powering large-scale pre-training. Next,
aiming to open-source a strong baseline for follow-up works, we developed the
multimodal algorithm that attacks different cinema/TV modelling problems with a
unified architecture. Extensive experiments on three cognitive-inspired tasks
yielded a number of novel observations (some of them being quite
counter-intuition), further validating the value of PTVD in promoting
multimodal research. The dataset and codes are released at
https://ptvd.github.io/.

该研究通过使用 1,106 集电视剧和 24,875 个由专业人士编写的信息密集的剧情句子，进行 449 个人类注释者的标注，构建了第一个面向剧情的电视领域的多模态数据集 PTVD，同时也是该领域中第一个非英文的数据集，并开发了多模态算法，针对不同的电影 / 电视建模问题进行处理，并在三个认知启发任务上进行了广泛的实验，从而验证了该数据集在促进多模态研究上的价值。

基于电视剧的大规模情节导向多模态数据集 PTVD

PTVD: A Large-Scale Plot-Oriented Multimodal Dataset Based on Television  Dramas

The increasing use of statistical data analysis in enterprise applications
has created an arms race among database vendors to offer ever more
sophisticated in-database analytics. One challenge in this race is that each
new statistical technique must be implemented from scratch in the RDBMS, which
leads to a lengthy and complex development process. We argue that the root
cause for this overhead is the lack of a unified architecture for in-database
analytics. Our main contribution in this work is to take a step towards such a
unified architecture. A key benefit of our unified architecture is that
performance optimizations for analytics techniques can be studied generically
instead of an ad hoc, per-technique fashion. In particular, our technical
contributions are theoretical and empirical studies of two key factors that we
found impact performance: the order data is stored, and parallelization of
computations on a single-node multicore RDBMS. We demonstrate the feasibility
of our architecture by integrating several popular analytics techniques into
two commercial and one open-source RDBMS. Our architecture requires changes to
only a few dozen lines of code to integrate a new statistical technique. We
then compare our approach with the native analytics tools offered by the
commercial RDBMSes on various analytics tasks, and validate that our approach
achieves competitive or higher performance, while still achieving the same
quality.

本文针对数据库供应商在提供日益复杂的 in-database analytics 功能时面临的开发难题进行探讨，提出了一种统一的架构来优化表现，其中包括对数据存储顺序和计算并行化的研究，并成功地将多种流行的分析技术集成到商业和开源 RDBMS 中，证明其具有与商业 RDBMS 本地工具相当或更高的性能。