Autoregressive and Masked Transformers are incredibly effective as generative
models and classifiers. While these models are most prevalent in NLP, they also
exhibit strong performance in other domains, such as vision. This work
contributes to the exploration of transformer-based models in synthetic data
generation for diverse application domains. In this paper, we present TabMT, a
novel Masked Transformer design for generating synthetic tabular data. TabMT
effectively addresses the unique challenges posed by heterogeneous data fields
and is natively able to handle missing data. Our design leverages improved
masking techniques to allow for generation and demonstrates state-of-the-art
performance from extremely small to extremely large tabular datasets. We
evaluate TabMT for privacy-focused applications and find that it is able to
generate high quality data with superior privacy tradeoffs.

本文介绍了 TabMT，一种用于生成合成表格数据的新型掩码 Transformer 设计，它有效应对异构数据字段带来的独特挑战，并能够处理缺失数据。我们评估了 TabMT 在以保护隐私为重点的应用中的性能，并发现它能够生成具有优越隐私权衡的高质量数据。

TabMT：使用掩码变换器生成表格数据

TabMT: Generating tabular data with masked transformers

Recommendation models are vital in delivering personalized user experiences
by leveraging the correlation between multiple input features. However, deep
learning-based recommendation models often face challenges due to evolving user
behaviour and item features, leading to covariate shifts. Effective
cross-feature learning is crucial to handle data distribution drift and
adapting to changing user behaviour. Traditional feature interaction techniques
have limitations in achieving optimal performance in this context.
This work introduces Ad-Rec, an advanced network that leverages feature
interaction techniques to address covariate shifts. This helps eliminate
irrelevant interactions in recommendation tasks. Ad-Rec leverages masked
transformers to enable the learning of higher-order cross-features while
mitigating the impact of data distribution drift. Our approach improves model
quality, accelerates convergence, and reduces training time, as measured by the
Area Under Curve (AUC) metric. We demonstrate the scalability of Ad-Rec and its
ability to achieve superior model quality through comprehensive ablation
studies.

本文介绍了一种名为 Ad-Rec 的高级网络，它利用特征交互技术来解决协变量偏移问题，在推荐任务中消除了不相关的交互，通过遮蔽变换器实现了学习高阶交叉特征，并减轻数据分布漂移的影响。我们的方法通过综合消融研究证明了 Ad-Rec 的可扩展性和卓越的模型质量，改进了模型质量，加快了收敛速度并减少了训练时间，这些都通过 AUC 度量得到了验证。

Ad-Rec: 处理推荐网络的协变量偏移的高级特征交互

Ad-Rec: Advanced Feature Interactions to Address Covariate-Shifts in  Recommendation Networks

We propose an efficient approach to train large diffusion models with masked
transformers. While masked transformers have been extensively explored for
representation learning, their application to generative learning is less
explored in the vision domain. Our work is the first to exploit masked training
to reduce the training cost of diffusion models significantly. Specifically, we
randomly mask out a high proportion (\emph{e.g.}, 50\%) of patches in diffused
input images during training. For masked training, we introduce an asymmetric
encoder-decoder architecture consisting of a transformer encoder that operates
only on unmasked patches and a lightweight transformer decoder on full patches.
To promote a long-range understanding of full patches, we add an auxiliary task
of reconstructing masked patches to the denoising score matching objective that
learns the score of unmasked patches. Experiments on ImageNet-256$\times$256
show that our approach achieves the same performance as the state-of-the-art
Diffusion Transformer (DiT) model, using only 31\% of its original training
time. Thus, our method allows for efficient training of diffusion models
without sacrificing the generative performance.

本论文提出了一种使用掩码 Transformer 训练大型扩散模型的高效方法，实现了在不牺牲生成性能的情况下，仅使用 31％的训练时间达到与最先进的扩散变压器模型相同性能的效果。