The accurate prediction of drug molecule solubility is essential for
determining their therapeutic effectiveness and safety, influencing the drug's
ADME processes. Traditional solubility prediction techniques often fail to
capture the complex nature of molecular tructures, leading to notable
deviations between predictions and actual results. For example, the Discussion
on Advanced Drug-Like Compound Structures. Lusci highlighted issues in
capturing crucial cyclic structural information in molecules with ring
structures. To overcome this issue, our research introduces a novel deep
learning framework combining attention-based transformers, Long Short-Term
Memory (LSTM) networks, and Graph Convolutional Networks (GCN), aimed at
enhancing the precision of solubility predictions. Utilizing a training set of
9,943 compounds and testing on an anticancer compound dataset, our method
achieved a correlation coefficient ($R^2$) of 0.55 and a Root Mean Square Error
(RMSE) of 0.59, which outperforms the benchmark models' scores of 0.52 ($R^2$)
and 0.61 (RMSE). Importantly, in an additional independent test, our model
significantly outperformed the baseline with an RMSE of 1.05 compared to 1.28,
a relative accuracy improvement of 45.9%. This research not only demonstrates
the vast potential of deep learning for improving solubility prediction
accuracy but also offers novel insights for drug design and selection in the
future. Continued efforts will be directed towards optimizing the model
architecture and extending its application to better support the drug
development process, underscoring the pivotal role of deep learning in drug
discovery.

通过引入深度学习框架，结合基于注意力机制的转换器、长短期记忆神经网络和图卷积网络，我们的研究旨在提高溶解度预测的准确性，为药物设计和选择提供新的见解和支持，为药物发现中的深度学习发挥重要作用展示了巨大潜力。

YZS 模型：基于图卷积网络和 Transformer-Attention 的有机药物溶解度预测模型

YZS-model: A Predictive Model for Organic Drug Solubility Based on Graph  Convolutional Networks and Transformer-Attention

Attention-based transformers have become the standard architecture in many
deep learning fields, primarily due to their ability to model long-range
dependencies and handle variable-length input sequences. However, the attention
mechanism with its quadratic complexity is a significant bottleneck in the
transformer architecture. This algorithm is only uni-directional in the decoder
and converges to a static pattern in over-parametrized decoder-only models. I
address this issue by developing a generative function as attention or
activation replacement. It still has the auto-regressive character by comparing
each token with the previous one. In my test setting with nanoGPT this yields a
smaller loss while having a smaller model. The loss further drops by
incorporating an average context vector. This concept of attention replacement
is distributed under the GNU AGPL v3 license at
this https URL

通过开发一种生成函数来解决基于关注机制的注意力变量效率低的问题，该方法在处理变长输入序列和建模长程依赖方面具有更小的模型和更小的损失。

突破注意力瓶颈

Breaking the Attention Bottleneck

Remote sensing image change captioning (RSICC) aims to automatically generate
sentences that describe content differences in remote sensing bitemporal
images. Recently, attention-based transformers have become a prevalent idea for
capturing the features of global change. However, existing transformer-based
RSICC methods face challenges, e.g., high parameters and high computational
complexity caused by the self-attention operation in the transformer encoder
component. To alleviate these issues, this paper proposes a Sparse Focus
Transformer (SFT) for the RSICC task. Specifically, the SFT network consists of
three main components, i.e. a high-level features extractor based on a
convolutional neural network (CNN), a sparse focus attention mechanism-based
transformer encoder network designed to locate and capture changing regions in
dual-temporal images, and a description decoder that embeds images and words to
generate sentences for captioning differences. The proposed SFT network can
reduce the parameter number and computational complexity by incorporating a
sparse attention mechanism within the transformer encoder network. Experimental
results on various datasets demonstrate that even with a reduction of over 90\%
in parameters and computational complexity for the transformer encoder, our
proposed network can still obtain competitive performance compared to other
state-of-the-art RSICC methods. The code can be available at

提出了一种稀疏聚焦变换器用于遥感图像变化编码任务，通过在变换器编码中引入稀疏注意机制，降低参数数量和计算复杂度，与其他最先进的遥感图像变化编码方法相比，仍能获得竞争性性能。

一种用于遥感图像变化描述的轻量级 Transformer

A Lightweight Transformer for Remote Sensing Image Change Captioning

In recent years, attention-based transformers have achieved tremendous
success across a variety of disciplines including natural languages. A key
ingredient behind their success is the generative pretraining procedure, during
which these models are trained on a large text corpus in an auto-regressive
manner. To shed light on this phenomenon, we propose a new framework that
allows both theory and systematic experiments to study the sequential modeling
capabilities of transformers through the lens of Markov chains. Inspired by the
Markovianity of natural languages, we model the data as a Markovian source and
utilize this framework to systematically study the interplay between the
data-distributional properties, the transformer architecture, the learnt
distribution, and the final model performance. In particular, we theoretically
characterize the loss landscape of single-layer transformers and show the
existence of global minima and bad local minima contingent upon the specific
data characteristics and the transformer architecture. Backed by experiments,
we demonstrate that our theoretical findings are in congruence with the
empirical results. We further investigate these findings in the broader context
of higher order Markov chains and deeper architectures, and outline open
problems in this arena. Code is available at
https://github.com/Bond1995/Markov.

通过马尔可夫链的角度研究变压器的序列建模能力，并在理论和实验上研究数据分布特性、变压器结构、学习分布和模型性能之间的相互作用。