The analysis of spatiotemporal data is increasingly utilized across diverse
domains, including transportation, healthcare, and meteorology. In real-world
settings, such data often contain missing elements due to issues like sensor
malfunctions and data transmission errors. The objective of spatiotemporal
imputation is to estimate these missing values by understanding the inherent
spatial and temporal relationships in the observed multivariate time series.
Traditionally, spatiotemporal imputation has relied on specific, intricate
architectures designed for this purpose, which suffer from limited
applicability and high computational complexity. In contrast, our approach
integrates pre-trained large language models (LLMs) into spatiotemporal
imputation, introducing a groundbreaking framework, GATGPT. This framework
merges a graph attention mechanism with LLMs. We maintain most of the LLM
parameters unchanged to leverage existing knowledge for learning temporal
patterns, while fine-tuning the upper layers tailored to various applications.
The graph attention component enhances the LLM's ability to understand spatial
relationships. Through tests on three distinct real-world datasets, our
innovative approach demonstrates comparable results to established deep
learning benchmarks.

在本研究中，我们提出了一种创新方法 GATGPT，它将已训练好的大型语言模型（LLM）与图注意机制相结合，用于时空插补。我们保持大部分 LLM 参数不变，以利用现有知识学习时间模式，并针对不同应用程序调整上层的微调。图注意组件增强了 LLM 理解空间关系的能力。通过对三个真实世界数据集的测试，我们的方法表现与已建立的深度学习基准相当。

GATGPT：基于图注意力网络的预训练大型语言模型用于时空填补

GATGPT: A Pre-trained Large Language Model with Graph Attention Network  for Spatiotemporal Imputation

Lossy gradient compression has become a practical tool to overcome the
communication bottleneck in centrally coordinated distributed training of
machine learning models. However, algorithms for decentralized training with
compressed communication over arbitrary connected networks have been more
complicated, requiring additional memory and hyperparameters. We introduce a
simple algorithm that directly compresses the model differences between
neighboring workers using low-rank linear compressors applied on model
differences. Inspired by the PowerSGD algorithm for centralized deep learning,
this algorithm uses power iteration steps to maximize the information
transferred per bit. We prove that our method requires no additional
hyperparameters, converges faster than prior methods, and is asymptotically
independent of both the network and the compression. Out of the box, these
compressors perform on par with state-of-the-art tuned compression algorithms
in a series of deep learning benchmarks.

介绍了一种使用低秩线性压缩器直接压缩相邻工作器之间的模型差异的简单算法，在分布式机器学习训练中具有实际应用价值，无需额外的超参数，收敛速度比之前的方法快，并且在一系列深度学习基准测试中的表现与经过调优的最先进压缩算法相当。