We introduce CrystalFormer, a transformer-based autoregressive model
specifically designed for space group-controlled generation of crystalline
materials. The space group symmetry significantly simplifies the crystal space,
which is crucial for data and compute efficient generative modeling of
crystalline materials. Leveraging the prominent discrete and sequential nature
of the Wyckoff positions, CrystalFormer learns to generate crystals by directly
predicting the species and locations of symmetry-inequivalent atoms in the unit
cell. Our results demonstrate that CrystalFormer matches state-of-the-art
performance on standard benchmarks for both validity, novelty, and stability of
the generated crystalline materials. Our analysis also shows that CrystalFormer
ingests sensible solid-state chemistry information from data for generative
modeling. The CrystalFormer unifies symmetry-based structure search and
generative pre-training in the realm of crystalline materials. The simplicity,
generality, and flexibility of CrystalFormer position it as a promising
architecture to be the foundational model of the entire crystalline materials
space, heralding a new era in materials modeling and discovery.

CrystalFormer 是一种基于 Transformer 的自回归模型，专门用于由空间群控制的晶体材料生成，其利用空间群对晶体空间进行了显著简化，通过直接预测单元胞中对称不等价原子的种类和位置来生成晶体，实验结果显示 CrystalFormer 在生成晶体材料的有效性、新颖性和稳定性方面与最先进的性能相匹配，并且可以从数据中准确地提取出合理的固态化学信息用于生成建模，它的简单性、通用性和灵活性使其成为整个晶体材料空间的基础模型，开启了材料建模和发现的新时代。

基于空间群信息的晶体材料生成变压器

Space Group Informed Transformer for Crystalline Materials Generation

Speech-driven 3D facial animation is challenging due to the complex geometry
of human faces and the limited availability of 3D audio-visual data. Prior
works typically focus on learning phoneme-level features of short audio windows
with limited context, occasionally resulting in inaccurate lip movements. To
tackle this limitation, we propose a Transformer-based autoregressive model,
FaceFormer, which encodes the long-term audio context and autoregressively
predicts a sequence of animated 3D face meshes. To cope with the data scarcity
issue, we integrate the self-supervised pre-trained speech representations.
Also, we devise two biased attention mechanisms well suited to this specific
task, including the biased cross-modal multi-head (MH) attention and the biased
causal MH self-attention with a periodic positional encoding strategy. The
former effectively aligns the audio-motion modalities, whereas the latter
offers abilities to generalize to longer audio sequences. Extensive experiments
and a perceptual user study show that our approach outperforms the existing
state-of-the-arts. The code will be made available.

提出了一种基于 Transformer 的自回归模型，称为 FaceFormer，用于声控 3D 面部动画。该模型能够对长时间音频上下文进行编码，并自回归预测一系列动画 3D 面网格。通过自监督预训练语音表示，使得该模型能够成功解决数据稀缺问题。同时，设计了两种有偏的注意机制，并进行为这个特定任务进行优化。大量实验证明，该方法优于现有的最先进技术。