Cell identity encompasses various semantic aspects of a cell, including cell
type, pathway information, disease information, and more, which are essential
for biologists to gain insights into its biological characteristics.
Understanding cell identity from the transcriptomic data, such as annotating
cell types, have become an important task in bioinformatics. As these semantic
aspects are determined by human experts, it is impossible for AI models to
effectively carry out cell identity understanding tasks without the supervision
signals provided by single-cell and label pairs. The single-cell pre-trained
language models (PLMs) currently used for this task are trained only on a
single modality, transcriptomics data, lack an understanding of cell identity
knowledge. As a result, they have to be fine-tuned for downstream tasks and
struggle when lacking labeled data with the desired semantic labels. To address
this issue, we propose an innovative solution by constructing a unified
representation of single-cell data and natural language during the pre-training
phase, allowing the model to directly incorporate insights related to cell
identity. More specifically, we introduce \textbf{LangCell}, the first
\textbf{Lang}uage-\textbf{Cell} pre-training framework. LangCell utilizes texts
enriched with cell identity information to gain a profound comprehension of
cross-modal knowledge. Results from experiments conducted on different
benchmarks show that LangCell is the only single-cell PLM that can work
effectively in zero-shot cell identity understanding scenarios, and also
significantly outperforms existing models in few-shot and fine-tuning cell
identity understanding scenarios.

LangCell 是第一个语言 - 细胞预训练框架，通过在预训练阶段构建单细胞数据和自然语言的统一表示，直接融合与细胞身份相关的见解，使其能在无监督细胞身份理解场景中有效工作，并在有监督和微调细胞身份理解场景中显著优于现有模型。

LangCell: 语言细胞预训练用于细胞身份理解

LangCell: Language-Cell Pre-training for Cell Identity Understanding

How to effectively represent molecules is a long-standing challenge for
molecular property prediction and drug discovery. This paper studies this
problem and proposes to incorporate chemical domain knowledge, specifically
related to chemical reactions, for learning effective molecular
representations. However, the inherent cross-modality property between chemical
reactions and molecules presents a significant challenge to address. To this
end, we introduce a novel method, namely MolKD, which Distills cross-modal
Knowledge in chemical reactions to assist Molecular property prediction.
Specifically, the reaction-to-molecule distillation model within MolKD
transfers cross-modal knowledge from a pre-trained teacher network learning
with one modality (i.e., reactions) into a student network learning with
another modality (i.e., molecules). Moreover, MolKD learns effective molecular
representations by incorporating reaction yields to measure transformation
efficiency of the reactant-product pair when pre-training on reactions.
Extensive experiments demonstrate that MolKD significantly outperforms various
competitive baseline models, e.g., 2.1% absolute AUC-ROC gain on Tox21. Further
investigations demonstrate that pre-trained molecular representations in MolKD
can distinguish chemically reasonable molecular similarities, which enables
molecular property prediction with high robustness and interpretability.

本文研究通过融入与化学反应相关的领域知识来学习有效的分子代表，并提出了一种新的方法 ——MolKD，通过将跨模态化学反应和分子之间的知识蒸馏为分子性质预测提供帮助。经过大量实验验证，该方法明显优于其他基线模型并具有较高的鲁棒性和可解释性。

MolKD: 化学反应跨模态知识蒸馏用于分子性质预测

MolKD: Distilling Cross-Modal Knowledge in Chemical Reactions for  Molecular Property Prediction

The success of deep learning heavily relies on large-scale data with
comprehensive labels, which is more expensive and time-consuming to fetch in 3D
compared to 2D images or natural languages. This promotes the potential of
utilizing models pretrained with data more than 3D as teachers for cross-modal
knowledge transferring. In this paper, we revisit masked modeling in a unified
fashion of knowledge distillation, and we show that foundational Transformers
pretrained with 2D images or natural languages can help self-supervised 3D
representation learning through training Autoencoders as Cross-Modal Teachers
(ACT). The pretrained Transformers are transferred as cross-modal 3D teachers
using discrete variational autoencoding self-supervision, during which the
Transformers are frozen with prompt tuning for better knowledge inheritance.
The latent features encoded by the 3D teachers are used as the target of masked
point modeling, wherein the dark knowledge is distilled to the 3D Transformer
students as foundational geometry understanding. Our ACT pretrained 3D learner
achieves state-of-the-art generalization capacity across various downstream
benchmarks, e.g., 88.21% overall accuracy on ScanObjectNN. Codes have been
released at this https URL

本研究使用自监督学习从 2D 图像和自然语言预训练 Transformers 的潜在特征教授 3D 表示学习，进一步促进了跨模态的知识转移和基础几何理解，实现了在各种现有测试中的最先进泛化能力。