The excellent generalization ability of self-supervised learning (SSL) for
speech foundation models has garnered significant attention. HuBERT is a
successful example that utilizes offline clustering to convert speech features
into discrete units for a masked language modeling pretext task. However,
simply clustering features as targets by k-means does not fully inspire the
model's performance. In this work, we present an unsupervised method to improve
SSL targets. Two models are proposed, MonoBERT and PolyBERT, which leverage
context-independent and context-dependent phoneme-based units for pre-training.
Our models outperform other SSL models significantly on the LibriSpeech
benchmark without the need for iterative re-clustering and re-training.
Furthermore, our models equipped with context-dependent units even outperform
target-improvement models that use labeled data during pre-training. How we
progressively improve the unit discovery process is demonstrated through
experiments.

本文提出了两个模型，MonoBERT 和 PolyBERT，它们使用上下文无关和上下文相关的音素为预训练提供了目标改进。我们的模型在 LibriSpeech 基准测试中显著优于其他自监督学习模型，无需迭代重新分簇和重新训练。此外，我们的模型配备上下文相关单元，甚至优于在预训练期间使用标记数据的目标改进模型。通过实验证明了如何逐步改进单元发现过程。

非监督单元发现在 SSL 语音表示方面的极限探索

Pushing the Limits of Unsupervised Unit Discovery for SSL Speech  Representation

Polymers are a vital part of everyday life. Their chemical universe is so
large that it presents unprecedented opportunities as well as significant
challenges to identify suitable application-specific candidates. We present a
complete end-to-end machine-driven polymer informatics pipeline that can search
this space for suitable candidates at unprecedented speed and accuracy. This
pipeline includes a polymer chemical fingerprinting capability called polyBERT
(inspired by Natural Language Processing concepts), and a multitask learning
approach that maps the polyBERT fingerprints to a host of properties. polyBERT
is a chemical linguist that treats the chemical structure of polymers as a
chemical language. The present approach outstrips the best presently available
concepts for polymer property prediction based on handcrafted fingerprint
schemes in speed by two orders of magnitude while preserving accuracy, thus
making it a strong candidate for deployment in scalable architectures including
cloud infrastructures.

一种快速高效的机器学习驱动的聚合物信息学管道，使用一种名为 polyBERT 的化学指纹能力来识别聚合物的化学结构，并将其映射到一系列物性。该管道通过 2 个数量级的速度优势远超过手工指纹方案，同时保持准确性，是部署在可扩展架构中的强有力选项。