In the domains of image and audio, diffusion models have shown impressive
performance. However, their application to discrete data types, such as
language, has often been suboptimal compared to autoregressive generative
models. This paper tackles the challenge of improving discrete diffusion models
by introducing a structured forward process that leverages the inherent
information hierarchy in discrete categories, such as words in text. Our
approach biases the generative process to produce certain categories before
others, resulting in a notable improvement in log-likelihood scores on the
text8 dataset. This work paves the way for more advances in discrete diffusion
models with potentially significant enhancements in performance.

通过引入结构化的前向过程，改进离散扩散模型，提高对离散数据类型的生成性能。

通过结构化偏好生成改进离散扩散模型

Improving Discrete Diffusion Models via Structured Preferential  Generation

Pre-trained transformer-based language models are becoming increasingly
popular due to their exceptional performance on various benchmarks. However,
concerns persist regarding the presence of hidden biases within these models,
which can lead to discriminatory outcomes and reinforce harmful stereotypes. To
address this issue, we propose Finspector, a human-centered visual inspection
tool designed to detect biases in different categories through log-likelihood
scores generated by language models. The goal of the tool is to enable
researchers to easily identify potential biases using visual analytics,
ultimately contributing to a fairer and more just deployment of these models in
both academic and industrial settings. Finspector is available at
this https URL

该研究提出了 Finspector, 一种基于视觉分析的人性化文本偏差检测工具，通过语言模型生成的对数似然得分来检测不同类别中的偏差，旨在促进公正使用这些模型的部署

Finspector：基于人的视角的视觉检查工具，用于探索和比较基础模型中的偏差

Finspector: A Human-Centered Visual Inspection Tool for Exploring and  Comparing Biases among Foundation Models

Matrix product states (MPS), a tensor network designed for one-dimensional
quantum systems, has been recently proposed for generative modeling of natural
data (such as images) in terms of `Born machine'. However, the exponential
decay of correlation in MPS restricts its representation power heavily for
modeling complex data such as natural images. In this work, we push forward the
effort of applying tensor networks to machine learning by employing the Tree
Tensor Network (TTN) which exhibits balanced performance in expressibility and
efficient training and sampling. We design the tree tensor network to utilize
the 2-dimensional prior of the natural images and develop sweeping learning and
sampling algorithms which can be efficiently implemented utilizing Graphical
Processing Units (GPU). We apply our model to random binary patterns and the
binary MNIST datasets of handwritten digits. We show that TTN is superior to
MPS for generative modeling in keeping correlation of pixels in natural images,
as well as giving better log-likelihood scores in standard datasets of
handwritten digits. We also compare its performance with state-of-the-art
generative models such as the Variational AutoEncoders, Restricted Boltzmann
machines, and PixelCNN. Finally, we discuss the future development of Tensor
Network States in machine learning problems.

本论文介绍了一种树张量网络 (TTN)，并将其应用于基于生成式建模的自然图像数据集以提高其可解释性，在 MNIST 数据集的表现中也获得更好的对数似然度得分。