Different fields in applied machine learning such as computer vision, speech
or natural language processing have been building domain-specialised solutions.
Currently, we are witnessing an opposing trend towards developing more
generalist architectures, driven by Large Language Models and multi-modal
foundational models. These architectures are designed to tackle a variety of
tasks, including those previously unseen and using inputs across multiple
modalities. Taking this trend of generalization to the extreme suggests the
possibility of a single deep network architecture capable of solving all tasks.
This position paper aims to explore developing such a unified architecture and
proposes a theoretical framework of how it could be constructed. Our proposal
is based on the following assumptions. Firstly, tasks are solved by following a
sequence of instructions, typically implemented in code for conventional
computing hardware, which inherently operates sequentially. Second, recent
Generative AI, especially Transformer-based models, demonstrate potential as an
architecture capable of constructing algorithms for a wide range of domains.
For example, GPT-4 shows exceptional capability at in-context learning of novel
tasks which is hard to explain in any other way than the ability to compose
novel solutions from fragments on previously learnt algorithms. Third, the
observation that the main missing component in developing a truly generalised
network is an efficient approach for self-consistent input of previously learnt
sub-steps of an algorithm and their (implicit) composition during the network's
internal forward pass. Our exploration delves into current capabilities and
limitations of Transformer-based and other methods in efficient and correct
algorithm composition and proposes a Transformer-like architecture as well as a
discrete learning framework to overcome these limitations.

提出一种理论框架构建统一的深度网络架构，基于 Transformer 模型以及离散学习框架，以解决当前构建通用网络的挑战。