Can conversational videos captured from multiple egocentric viewpoints reveal
the map of a scene in a cost-efficient way? We seek to answer this question by
proposing a new problem: efficiently building the map of a previously unseen 3D
environment by exploiting shared information in the egocentric audio-visual
observations of participants in a natural conversation. Our hypothesis is that
as multiple people ("egos") move in a scene and talk among themselves, they
receive rich audio-visual cues that can help uncover the unseen areas of the
scene. Given the high cost of continuously processing egocentric visual
streams, we further explore how to actively coordinate the sampling of visual
information, so as to minimize redundancy and reduce power use. To that end, we
present an audio-visual deep reinforcement learning approach that works with
our shared scene mapper to selectively turn on the camera to efficiently chart
out the space. We evaluate the approach using a state-of-the-art audio-visual
simulator for 3D scenes as well as real-world video. Our model outperforms
previous state-of-the-art mapping methods, and achieves an excellent
cost-accuracy tradeoff. Project: this http URL

利用多个自我视角的、来自多方的对话视频，结合深度强化学习的方法，高效地构建出未知 3D 环境的地图，达到了出色的性价比，优于以往最先进的方法。

Chat2Map：多自我对话中高效的场景映射

Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations

Emotion Recognition in Conversations (ERC) is crucial in developing
sympathetic human-machine interaction. In conversational videos, emotion can be
present in multiple modalities, i.e., audio, video, and transcript. However,
due to the inherent characteristics of these modalities, multi-modal ERC has
always been considered a challenging undertaking. Existing ERC research focuses
mainly on using text information in a discussion, ignoring the other two
modalities. We anticipate that emotion recognition accuracy can be improved by
employing a multi-modal approach. Thus, in this study, we propose a Multi-modal
Fusion Network (M2FNet) that extracts emotion-relevant features from visual,
audio, and text modality. It employs a multi-head attention-based fusion
mechanism to combine emotion-rich latent representations of the input data. We
introduce a new feature extractor to extract latent features from the audio and
visual modality. The proposed feature extractor is trained with a novel
adaptive margin-based triplet loss function to learn emotion-relevant features
from the audio and visual data. In the domain of ERC, the existing methods
perform well on one benchmark dataset but not on others. Our results show that
the proposed M2FNet architecture outperforms all other methods in terms of
weighted average F1 score on well-known MELD and IEMOCAP datasets and sets a
new state-of-the-art performance in ERC.

本篇研究利用多模态方法提高情感识别的准确性，提出了一种多模态融合网络 M2FNet，利用视觉、音频和文本模态提取情感相关特征，并使用多头注意力机制进行融合。经实验验证，该方法在 MELD 和 IEMOCAP 数据集上表现显著优于其他方法，达到了情感识别领域的最新性能表现。