Attention-based transformers have become the standard architecture in many
deep learning fields, primarily due to their ability to model long-range
dependencies and handle variable-length input sequences. However, the attention
mechanism with its quadratic complexity is a significant bottleneck in the
transformer architecture. This algorithm is only uni-directional in the decoder
and converges to a static pattern in over-parametrized decoder-only models. I
address this issue by developing a generative function as attention or
activation replacement. It still has the auto-regressive character by comparing
each token with the previous one. In my test setting with nanoGPT this yields a
smaller loss while having a smaller model. The loss further drops by
incorporating an average context vector. This concept of attention replacement
is distributed under the GNU AGPL v3 license at
this https URL

通过开发一种生成函数来解决基于关注机制的注意力变量效率低的问题，该方法在处理变长输入序列和建模长程依赖方面具有更小的模型和更小的损失。

突破注意力瓶颈

Breaking the Attention Bottleneck

Compressed sensing (CS) is about recovering a structured signal from its
under-determined linear measurements. Starting from sparsity, recovery methods
have steadily moved towards more complex structures. Emerging machine learning
tools such as generative functions that are based on neural networks are able
to learn general complex structures from training data. This makes them
potentially powerful tools for designing CS algorithms. Consider a desired
class of signals $\cal Q$, ${\cal Q}\subset{R}^n$, and a corresponding
generative function $g:{\cal U}^k\rightarrow {R}^n$, ${\cal U}\subset {R}$,
such that $\sup_{{\bf x}\in {\cal Q}}\min_{{\bf u}\in{\cal U}^k}{1\over
\sqrt{n}}\|g({\bf u})-{\bf x}\|\leq \delta$. A recovery method based on $g$
seeks $g({\bf u})$ with minimum measurement error. In this paper, the
performance of such a recovery method is studied, under both noisy and
noiseless measurements. In the noiseless case, roughly speaking, it is proven
that, as $k$ and $n$ grow without bound and $\delta$ converges to zero, if the
number of measurements ($m$) is larger than the input dimension of the
generative model ($k$), then asymptotically, almost lossless recovery is
possible. Furthermore, the performance of an efficient iterative algorithm
based on projected gradient descent is studied. In this case, an auto-encoder
is used to define and enforce the source structure at the projection step. The
auto-encoder is defined by encoder and decoder (generative) functions
$f:{R}^n\to{\cal U}^k$ and $g:{\cal U}^k\to{R}^n$, respectively. We
theoretically prove that, roughly, given $m>40k\log{1\over \delta}$
measurements, such an algorithm converges to the vicinity of the desired
result, even in the presence of additive white Gaussian noise. Numerical
results exploring the effectiveness of the proposed method are presented.

通过生成函数和迭代算法解决压缩感知问题，研究了在噪声存在的情况下恢复具有复杂结构信号的表现，并使用自动编码器定义和强制源结构.

使用自编码器解决反问题

Solving inverse problems via auto-encoders

We study the problem of transferring a sample in one domain to an analog
sample in another domain. Given two related domains, S and T, we would like to
learn a generative function G that maps an input sample from S to the domain T,
such that the output of a given function f, which accepts inputs in either
domains, would remain unchanged. Other than the function f, the training data
is unsupervised and consist of a set of samples from each domain. The Domain
Transfer Network (DTN) we present employs a compound loss function that
includes a multiclass GAN loss, an f-constancy component, and a regularizing
component that encourages G to map samples from T to themselves. We apply our
method to visual domains including digits and face images and demonstrate its
ability to generate convincing novel images of previously unseen entities,
while preserving their identity.

本文介绍了一种 “Domain Transfer Network (DTN)” 算法，它可以通过学习生成一个映射函数，将一个领域中的样本转移到另一个领域，并能在保持其他函数不变的情况下，在包含多分类 GAN 损失的复合损失函数的指导下，增强互补函数，鼓励映射函数将样本从 T 到 T 自身，并且用于数字和面部图像等领域生成新的可信图像，同时保留它们的身份.