Sora unveils the potential of scaling Diffusion Transformer for generating
photorealistic images and videos at arbitrary resolutions, aspect ratios, and
durations, yet it still lacks sufficient implementation details. In this
technical report, we introduce the Lumina-T2X family - a series of Flow-based
Large Diffusion Transformers (Flag-DiT) equipped with zero-initialized
attention, as a unified framework designed to transform noise into images,
videos, multi-view 3D objects, and audio clips conditioned on text
instructions. By tokenizing the latent spatial-temporal space and incorporating
learnable placeholders such as [nextline] and [nextframe] tokens, Lumina-T2X
seamlessly unifies the representations of different modalities across various
spatial-temporal resolutions. This unified approach enables training within a
single framework for different modalities and allows for flexible generation of
multimodal data at any resolution, aspect ratio, and length during inference.
Advanced techniques like RoPE, RMSNorm, and flow matching enhance the
stability, flexibility, and scalability of Flag-DiT, enabling models of
Lumina-T2X to scale up to 7 billion parameters and extend the context window to
128K tokens. This is particularly beneficial for creating ultra-high-definition
images with our Lumina-T2I model and long 720p videos with our Lumina-T2V
model. Remarkably, Lumina-T2I, powered by a 5-billion-parameter Flag-DiT,
requires only 35% of the training computational costs of a
600-million-parameter naive DiT. Our further comprehensive analysis underscores
Lumina-T2X's preliminary capability in resolution extrapolation,
high-resolution editing, generating consistent 3D views, and synthesizing
videos with seamless transitions. We expect that the open-sourcing of
Lumina-T2X will further foster creativity, transparency, and diversity in the
generative AI community.

Lumina-T2X 是一种 Flow-based Large Diffusion Transformer (Flag-DiT) 模型，通过将不同模态的表示统一到一个框架中，实现了任意分辨率、宽高比和长度的图像、视频、多视角 3D 物体和音频剪辑的生成。采用 RoPE、RMSNorm 和 Flow matching 等先进技术，Lumina-T2X 模型在稳定性、灵活性和可伸缩性方面取得了突破，并在分辨率外推、高分辨率编辑、一致的 3D 视图合成和视频过渡等任务中展现了出色的性能。

Lumina-T2X：基于流式大规模扩散转换器将文本转化为任何模态、分辨率和持续时间

Lumina-T2X: Transforming Text into Any Modality, Resolution, and  Duration via Flow-based Large Diffusion Transformers

In cluster analysis, a common first step is to scale the data aiming to
better partition them into clusters. Even though many different techniques have
throughout many years been introduced to this end, it is probably fair to say
that the workhorse in this preprocessing phase has been to divide the data by
the standard deviation along each dimension. Like division by the standard
deviation, the great majority of scaling techniques can be said to have roots
in some sort of statistical take on the data. Here we explore the use of
multidimensional shapes of data, aiming to obtain scaling factors for use prior
to clustering by some method, like k-means, that makes explicit use of
distances between samples. We borrow from the field of cosmology and related
areas the recently introduced notion of shape complexity, which in the variant
we use is a relatively simple, data-dependent nonlinear function that we show
can be used to help with the determination of appropriate scaling factors.
Focusing on what might be called "midrange" distances, we formulate a
constrained nonlinear programming problem and use it to produce candidate
scaling-factor sets that can be sifted on the basis of further considerations
of the data, say via expert knowledge. We give results on some iconic data
sets, highlighting the strengths and potential weaknesses of the new approach.
These results are generally positive across all the data sets used.

利用多维数据的形状复杂度概念，应用在特定的非线性函数上，制定了一种新的 “中等距离” 下的基于约束的非线性规划问题，从而探索更有效的聚类缩放因子数量的方法。

聚类分析中的形状复杂性

Shape complexity in cluster analysis

This paper presents an in-depth investigation on integrating neural language
models in translation systems. Scaling neural language models is a difficult
task, but crucial for real-world applications. This paper evaluates the impact
on end-to-end MT quality of both new and existing scaling techniques. We show
when explicitly normalising neural models is necessary and what optimisation
tricks one should use in such scenarios. We also focus on scalable training
algorithms and investigate noise contrastive estimation and diagonal contexts
as sources for further speed improvements. We explore the trade-offs between
neural models and back-off n-gram models and find that neural models make
strong candidates for natural language applications in memory constrained
environments, yet still lag behind traditional models in raw translation
quality. We conclude with a set of recommendations one should follow to build a
scalable neural language model for MT.

该研究论文深入探讨了集成神经语言模型在翻译系统中的应用，评估了多种扩展神经模型的技术对最终翻译质量的影响，并提出了一系列训练算法、优化技巧和建议，以构建一个适用于机器翻译的可扩展的神经语言模型。