Wireless communications at high-frequency bands with large antenna arrays
face challenges in beam management, which can potentially be improved by
multimodality sensing information from cameras, LiDAR, radar, and GPS. In this
paper, we present a multimodal transformer deep learning framework for
sensing-assisted beam prediction. We employ a convolutional neural network to
extract the features from a sequence of images, point clouds, and radar raw
data sampled over time. At each convolutional layer, we use transformer
encoders to learn the hidden relations between feature tokens from different
modalities and time instances over abstraction space and produce encoded
vectors for the next-level feature extraction. We train the model on a
combination of different modalities with supervised learning. We try to enhance
the model over imbalanced data by utilizing focal loss and exponential moving
average. We also evaluate data processing and augmentation techniques such as
image enhancement, segmentation, background filtering, multimodal data
flipping, radar signal transformation, and GPS angle calibration. Experimental
results show that our solution trained on image and GPS data produces the best
distance-based accuracy of predicted beams at 78.44%, with effective
generalization to unseen day scenarios near 73% and night scenarios over 84%.
This outperforms using other modalities and arbitrary data processing
techniques, which demonstrates the effectiveness of transformers with feature
fusion in performing radio beam prediction from images and GPS. Furthermore,
our solution could be pretrained from large sequences of multimodality wireless
data, on fine-tuning for multiple downstream radio network tasks.

本文介绍了一种多模态变压器深度学习框架，用于辅助感知的波束预测。通过使用卷积神经网络从一系列图像、点云和雷达原始数据中提取特征，并使用变压器编码器学习不同模态和时间实例之间的隐藏关系，生成下一级特征提取的编码向量。通过在不同模态的组合上进行监督学习来训练模型，实验结果表明，使用图像和 GPS 数据训练的解决方案在预测波束的距离准确率方面表现最佳，为 78.44％，并具有有效的推广性。