We propose Token Turing Machines (TTM), a sequential, autoregressive
Transformer model with memory for real-world sequential visual understanding.
Our model is inspired by the seminal Neural Turing Machine, and has an external
memory consisting of a set of tokens which summarise the previous history
(i.e., frames). This memory is efficiently addressed, read and written using a
Transformer as the processing unit/controller at each step. The model's memory
module ensures that a new observation will only be processed with the contents
of the memory (and not the entire history), meaning that it can efficiently
process long sequences with a bounded computational cost at each step. We show
that TTM outperforms other alternatives, such as other Transformer models
designed for long sequences and recurrent neural networks, on two real-world
sequential visual understanding tasks: online temporal activity detection from
videos and vision-based robot action policy learning.
Code is publicly available at:
this https URL

提出了 Token Turing Machines (TTM)，这是一种用于实现现实生活中序列视觉理解的序列自回归 Transformer 模型，它具有外部记忆，可以高效地处理长序列，其内存模块可以确保新的观察仅与记忆内容一起处理 (而不是整个历史记录)，并在两个实际任务上比其他替代方案表现更好。