The recent advancement of pre-trained Transformer models has propelled the
development of effective text mining models across various biomedical tasks.
However, these models are primarily learned on the textual data and often lack
the domain knowledge of the entities to capture the context beyond the
sentence. In this study, we introduced a novel framework that enables the model
to learn multi-omnics biological information about entities (proteins) with the
help of additional multi-modal cues like molecular structure. Towards this,
rather developing modality-specific architectures, we devise a generalized and
optimized graph based multi-modal learning mechanism that utilizes the
GraphBERT model to encode the textual and molecular structure information and
exploit the underlying features of various modalities to enable end-to-end
learning. We evaluated our proposed method on ProteinProtein Interaction task
from the biomedical corpus, where our proposed generalized approach is observed
to be benefited by the additional domain-specific modality.

本研究介绍了一种使用多模态线索（如分子结构）帮助模型学习实体（蛋白质）的多组学生物信息的新颖框架，以图形为基础的多模态学习机制利用 GraphBERT 模型编码了文本和分子结构信息，并利用不同模态的基础特征进行端到端的学习。在生物医学领域的蛋白质相互作用任务中，我们的提出的广义方法发现受到了领域特定模态的额外获益。

多模态基于图的转换器框架用于生物医学关系抽取

Multimodal Graph-based Transformer Framework for Biomedical Relation  Extraction

Active speaker detection requires a solid integration of multi-modal cues.
While individual modalities can approximate a solution, accurate predictions
can only be achieved by explicitly fusing the audio and visual features and
modeling their temporal progression. Despite its inherent muti-modal nature,
current methods still focus on modeling and fusing short-term audiovisual
features for individual speakers, often at frame level. In this paper we
present a novel approach to active speaker detection that directly addresses
the multi-modal nature of the problem, and provides a straightforward strategy
where independent visual features from potential speakers in the scene are
assigned to a previously detected speech event. Our experiments show that, an
small graph data structure built from a single frame, allows to approximate an
instantaneous audio-visual assignment problem. Moreover, the temporal extension
of this initial graph achieves a new state-of-the-art on the AVA-ActiveSpeaker
dataset with a mAP of 88.8\%.

本文提出了将图形数据结构应用于音频和视觉信号融合的新方法，将多模式的音视频特征分配给先前检测到的语音事件，实现了 88.8％的 AV-ActiveSpeaker 数据集性能最优。

MAAS: 多模态分配用于活跃说话人检测

MAAS: Multi-modal Assignation for Active Speaker Detection

In this paper, we introduce our submissions for the tasks of trimmed activity
recognition (Kinetics) and trimmed event recognition (Moments in Time) for
Activitynet Challenge 2018. In the two tasks, non-local neural networks and
temporal segment networks are implemented as our base models. Multi-modal cues
such as RGB image, optical flow and acoustic signal have also been used in our
method. We also propose new non-local-based models for further improvement on
the recognition accuracy. The final submissions after ensembling the models
achieve 83.5% top-1 accuracy and 96.8% top-5 accuracy on the Kinetics
validation set, 35.81% top-1 accuracy and 62.59% top-5 accuracy on the MIT
validation set.

本篇论文介绍我们针对 Activitynet Challenge 2018 中的两个任务 ——trimmed activity recognition (Kinetics) 和 trimmed event recognition (Moments in Time)，所采用的基于 non-local 神经网络和 temporal segment 网络的多模态方法，并使用新型的 non-local-based 模型以提高识别准确性，最终我们通过模型融合在 Kinetics 验证集中取得了 83.5% 的 top-1 准确度和 96.8% 的 top-5 准确度，在 MIT 验证集中则取得了 35.81% 的 top-1 准确度和 62.59% 的 top-5 准确度。