Time-series analysis plays a pivotal role across a range of critical
applications, from finance to healthcare, which involves various tasks, such as
forecasting and classification. To handle the inherent complexities of
time-series data, such as high dimensionality and noise, traditional supervised
learning methods first annotate extensive labels for time-series data in each
task, which is very costly and impractical in real-world applications. In
contrast, pre-trained foundation models offer a promising alternative by
leveraging unlabeled data to capture general time series patterns, which can
then be fine-tuned for specific tasks. However, existing approaches to
pre-training such models typically suffer from high-bias and low-generality
issues due to the use of predefined and rigid augmentation operations and
domain-specific data training. To overcome these limitations, this paper
introduces UniCL, a universal and scalable contrastive learning framework
designed for pretraining time-series foundation models across cross-domain
datasets. Specifically, we propose a unified and trainable time-series
augmentation operation to generate pattern-preserved, diverse, and low-bias
time-series data by leveraging spectral information. Besides, we introduce a
scalable augmentation algorithm capable of handling datasets with varying
lengths, facilitating cross-domain pretraining. Extensive experiments on two
benchmark datasets across eleven domains validate the effectiveness of UniCL,
demonstrating its high generalization on time-series analysis across various
fields.

UniCL 引入了一种通用的和可扩展的对比学习框架，用于预训练时间序列基础模型，跨领域数据集验证了 UniCL 在各个领域的高泛化性能。

UniCL：大型时间序列模型的通用对比学习框架

UniCL: A Universal Contrastive Learning Framework for Large Time Series  Models

Traditionally, large language models have been either trained on general web
crawls or domain-specific data. However, recent successes of generative large
language models, have shed light on the benefits of cross-domain datasets. To
examine the significance of prioritizing data diversity over quality, we
present a German dataset comprising texts from five domains, along with another
dataset aimed at containing high-quality data. Through training a series of
models ranging between 122M and 750M parameters on both datasets, we conduct a
comprehensive benchmark on multiple downstream tasks. Our findings demonstrate
that the models trained on the cross-domain dataset outperform those trained on
quality data alone, leading to improvements up to $4.45\%$ over the previous
state-of-the-art. The models are available at
this https URL

通过使用不同领域的语料库和不同质量的数据，我们训练了一系列的大语言模型，在多个下游任务中，跨领域数据集训练的模型表现优于仅使用高质量数据训练的模型，前者的性能提升高达 4.45%。

论跨领域数据对德语语言模型的影响

On the Impact of Cross-Domain Data on German Language Models

Knowledge distillation has recently become popular as a method of model
aggregation on the server for federated learning. It is generally assumed that
there are abundant public unlabeled data on the server. However, in reality,
there exists a domain discrepancy between the datasets of the server domain and
a client domain, which limits the performance of knowledge distillation. How to
improve the aggregation under such a domain discrepancy setting is still an
open problem. In this paper, we first analyze the generalization bound of the
aggregation model produced from knowledge distillation for the client domains,
and then describe two challenges, server-to-client discrepancy and
client-to-client discrepancy, brought to the aggregation model by the domain
discrepancies. Following our analysis, we propose an adaptive knowledge
aggregation algorithm FedD3A based on domain discrepancy aware distillation to
lower the bound. FedD3A performs adaptive weighting at the sample level in each
round of FL. For each sample in the server domain, only the client models of
its similar domains will be selected for playing the teacher role. To achieve
this, we show that the discrepancy between the server-side sample and the
client domain can be approximately measured using a subspace projection matrix
calculated on each client without accessing its raw data. The server can thus
leverage the projection matrices from multiple clients to assign weights to the
corresponding teacher models for each server-side sample. We validate FedD3A on
two popular cross-domain datasets and show that it outperforms the compared
competitors in both cross-silo and cross-device FL settings.

本文针对 Federated Learning 领域中存在的不同数据集之间的 Domain Discrepancy 挑战，分析了来源于知识蒸馏模型的聚合模型的泛化界限，并提出了一种基于 Domain Discrepancy Aware Distillation 的自适应知识聚合算法 FedD3A 来降低该界限，并在两个流行的跨域数据集上验证了该算法的有效性。

领域差异感知蒸馏：联邦学习模型聚合

Domain Discrepancy Aware Distillation for Model Aggregation in Federated Learning

Past work in relation extraction mostly focuses on binary relation between
entity pairs within single sentence. Recently, the NLP community has gained
interest in relation extraction in entity pairs spanning multiple sentences. In
this paper, we propose a novel architecture for this task: inter-sentential
dependency-based neural networks (iDepNN). iDepNN models the shortest and
augmented dependency paths via recurrent and recursive neural networks to
extract relationships within (intra-) and across (inter-) sentence boundaries.
Compared to SVM and neural network baselines, iDepNN is more robust to false
positives in relationships spanning sentences.
We evaluate our models on four datasets from newswire (MUC6) and medical
(BioNLP shared task) domains that achieve state-of-the-art performance and show
a better balance in precision and recall for inter-sentential relationships. We
perform better than 11 teams participating in the BioNLP shared task 2016 and
achieve a gain of 5.2% (0.587 vs 0.558) in F1 over the winning team. We also
release the crosssentence annotations for MUC6.

本文提出了一种新的架构 iDepNN，它可以通过循环和递归神经网络对最短和增强的依赖路径进行建模，以从跨句子的实体对中提取关系，并在新闻和医学领域的数据集上实现了最新技术的表现和更好的精度和召回的平衡，在 BioNLP 共享任务 2016 中胜过了 11 个参赛队伍，并获得了 5.2％的 F1 增益。