In the fields of brain-computer interaction and cognitive neuroscience,
effective decoding of auditory signals from task-based functional magnetic
resonance imaging (fMRI) is key to understanding how the brain processes
complex auditory information. Although existing methods have enhanced decoding
capabilities, limitations remain in information utilization and model
representation. To overcome these challenges, we propose an innovative
multi-task learning model, Physics-informed Embedding Network with Multi-Task
Transformer (PEMT-Net), which enhances decoding performance through
physics-informed embedding and deep learning techniques. PEMT-Net consists of
two principal components: feature augmentation and classification. For feature
augmentation, we propose a novel approach by creating neural embedding graphs
via node embedding, utilizing random walks to simulate the physical diffusion
of neural information. This method captures both local and non-local
information overflow and proposes a position encoding based on relative
physical coordinates. In the classification segment, we propose adaptive
embedding fusion to maximally capture linear and non-linear characteristics.
Furthermore, we propose an innovative parameter-sharing mechanism to optimize
the retention and learning of extracted features. Experiments on a specific
dataset demonstrate PEMT-Net's significant performance in multi-task auditory
signal decoding, surpassing existing methods and offering new insights into the
brain's mechanisms for processing complex auditory information.

通过物理启发嵌入网络和深度学习技术，我们提出了一种创新的多任务学习模型 PEMT-Net，用于增强多任务听觉信号解码性能，从而超越现有方法并为理解大脑处理复杂听觉信息的机制提供新的见解。

通过物理知识引导的多任务转换器的嵌入网络理解听觉诱发的脑信号

Understanding Auditory Evoked Brain Signal via Physics-informed  Embedding Network with Multi-Task Transformer

The analysis of the current integration attempts of some modes and use cases
of user-machine interaction is presented. The new concept of the user-driven
intelligent interface is proposed on the basis of multimodal augmented reality
and brain-computer interaction for various applications: in disabilities
studies, education, home care, health care, etc. The several use cases of
multimodal augmentation are presented. The perspectives of the better human
comprehension by the immediate feedback through neurophysical channels by means
of brain-computer interaction are outlined. It is shown that brain-computer
interface (BCI) technology provides new strategies to overcome limits of the
currently available user interfaces, especially for people with functional
disabilities. The results of the previous studies of the low end consumer and
open-source BCI-devices allow us to conclude that combination of machine
learning (ML), multimodal interactions (visual, sound, tactile) with BCI will
profit from the immediate feedback from the actual neurophysical reactions
classified by ML methods. In general, BCI in combination with other modes of AR
interaction can deliver much more information than these types of interaction
themselves. Even in the current state the combined AR-BCI interfaces could
provide the highly adaptable and personal services, especially for people with
functional disabilities.

本文提出了用户驱动的智能界面的新概念，它应用于各种领域中，如残疾研究，教育，家庭护理，医疗保健等，通过脑机交互等方式，利用多模态增强现实技术，以及通过机器学习等方法，分类实时神经物理反应的即时反馈，以克服当前可用用户界面的局限性，特别是针对功能残疾人群。即使在当前状态下，结合增强现实和脑机交互的接口也能提供高度适应性和个性化的服务。

基于多模态增强现实和脑机交互的面向功能障碍人群的用户驱动智能界面

User-driven Intelligent Interface on the Basis of Multimodal Augmented  Reality and Brain-Computer Interaction for People with Functional  Disabilities