In the current landscape of foundation model training, there is a significant
reliance on public domain data, which is nearing exhaustion according to recent
research. To further scale up, it is crucial to incorporate collaboration among
multiple specialized and high-quality private domain data sources. However, the
challenge of training models locally without sharing private data presents
numerous obstacles in data quality control. To tackle this issue, we propose a
data quality control pipeline for federated fine-tuning of foundation models.
This pipeline computes scores reflecting the quality of training data and
determines a global threshold for a unified standard, aiming for improved
global performance. Our experiments show that the proposed quality control
pipeline facilitates the effectiveness and reliability of the model training,
leading to better performance.

当前基础模型训练的情况表明，公共领域数据已接近枯竭，因此需要在多个专门化和高质量的私有领域数据源之间加强合作。为了解决本地训练模型而不共享私有数据所带来的质量控制问题，我们提出了一个针对基础模型的联邦微调的数据质量控制流程。该流程通过计算反映训练数据质量的分数，并确定一个统一标准的全局阈值，旨在提高整体性能。我们的实验证明，所提出的质量控制流程有助于模型训练的效果和可靠性，从而提高性能。

增强基础模型联邦微调中的数据质量

Enhancing Data Quality in Federated Fine-Tuning of Foundation Models

Storing and streaming high dimensional data for foundation model training
became a critical requirement with the rise of foundation models beyond natural
language. In this paper we introduce TensorBank, a petabyte scale tensor
lakehouse capable of streaming tensors from Cloud Object Store (COS) to GPU
memory at wire speed based on complex relational queries. We use Hierarchical
Statistical Indices (HSI) for query acceleration. Our architecture allows to
directly address tensors on block level using HTTP range reads. Once in GPU
memory, data can be transformed using PyTorch transforms. We provide a generic
PyTorch dataset type with a corresponding dataset factory translating
relational queries and requested transformations as an instance. By making use
of the HSI, irrelevant blocks can be skipped without reading them as those
indices contain statistics on their content at different hierarchical
resolution levels. This is an opinionated architecture powered by open
standards and making heavy use of open-source technology. Although, hardened
for production use using geospatial-temporal data, this architecture
generalizes to other use case like computer vision, computational neuroscience,
biological sequence analysis and more.

基于复杂的关系查询，TensorBank 是一个能够从云对象存储流式传输张量到 GPU 内存的 PB 级张量湖，使用 HSI 进行查询加速，并且可以使用 PyTorch 转换对数据进行处理，该架构适用于各种用例包括自然语言处理、计算机视觉、计算神经科学和生物序列分析等。