Nutrition estimation is crucial for effective dietary management and overall
health and well-being. Existing methods often struggle with sub-optimal
accuracy and can be time-consuming. In this paper, we propose NuNet, a
transformer-based network designed for nutrition estimation that utilizes both
RGB and depth information from food images. We have designed and implemented a
multi-scale encoder and decoder, along with two types of feature fusion
modules, specialized for estimating five nutritional factors. These modules
effectively balance the efficiency and effectiveness of feature extraction with
flexible usage of our customized attention mechanisms and fusion strategies.
Our experimental study shows that NuNet outperforms its variants and existing
solutions significantly for nutrition estimation. It achieves an error rate of
15.65%, the lowest known to us, largely due to our multi-scale architecture and
fusion modules. This research holds practical values for dietary management
with huge potential for transnational research and deployment and could inspire
other applications involving multiple data types with varying degrees of
importance.

NuNet 是一种基于变压器的网络，利用食物图像的 RGB 和深度信息进行营养估计，通过多尺度架构和融合模块，实现了最低已知的 15.65% 的误差率，在饮食管理方面具有重要实用价值和跨国研究和部署的潜力。

膳食管理的营养估计：基于深度感知的变形器方法

Nutrition Estimation for Dietary Management: A Transformer Approach with  Depth Sensing

The integration of different modalities, such as audio and visual
information, plays a crucial role in human perception of the surrounding
environment. Recent research has made significant progress in designing fusion
modules for audio-visual speech separation. However, they predominantly focus
on multi-modal fusion architectures situated either at the top or bottom
positions, rather than comprehensively considering multi-modal fusion at
various hierarchical positions within the network. In this paper, we propose a
novel model called self- and cross-attention network (SCANet), which leverages
the attention mechanism for efficient audio-visual feature fusion. SCANet
consists of two types of attention blocks: self-attention (SA) and
cross-attention (CA) blocks, where the CA blocks are distributed at the top
(TCA), middle (MCA) and bottom (BCA) of SCANet. These blocks maintain the
ability to learn modality-specific features and enable the extraction of
different semantics from audio-visual features. Comprehensive experiments on
three standard audio-visual separation benchmarks (LRS2, LRS3, and VoxCeleb2)
demonstrate the effectiveness of SCANet, outperforming existing
state-of-the-art (SOTA) methods while maintaining comparable inference time.

提出了一种名为自注意力网络（SCANet）的模型，通过利用注意机制实现有效的音频 - 视觉特征融合，其中包含了自注意力块（SA）和交叉注意力块（CA），能够从音频 - 视觉特征中提取不同的语义信息，并在三个标准音频 - 视觉分离基准上得到了更好的效果。

SCANet: 自我和交叉注意网络用于音视频语音分离

SCANet: A Self- and Cross-Attention Network for Audio-Visual Speech  Separation

Most existing RGB-based trackers target low frame rate benchmarks of around
30 frames per second. This setting restricts the tracker's functionality in the
real world, especially for fast motion. Event-based cameras as bioinspired
sensors provide considerable potential for high frame rate tracking due to
their high temporal resolution. However, event-based cameras cannot offer
fine-grained texture information like conventional cameras. This unique
complementarity motivates us to combine conventional frames and events for high
frame rate object tracking under various challenging conditions. Inthispaper,
we propose an end-to-end network consisting of multi-modality alignment and
fusion modules to effectively combine meaningful information from both
modalities at different measurement rates. The alignment module is responsible
for cross-style and cross-frame-rate alignment between frame and event
modalities under the guidance of the moving cues furnished by events. While the
fusion module is accountable for emphasizing valuable features and suppressing
noise information by the mutual complement between the two modalities.
Extensive experiments show that the proposed approach outperforms
state-of-the-art trackers by a significant margin in high frame rate tracking.
With the FE240hz dataset, our approach achieves high frame rate tracking up to
240Hz.

该论文提出了使用多模方式结合 RGB-based trackers 和 event-based cameras 进行 high frame rate tracking 的方法，并应用 multi-modality alignment 和 fusion modules 对两种模式的信息进行融合。在 FE240hz 数据集上，该方法表现优异，实现了高达 240Hz 的高帧率追踪。

高帧率跟踪的帧 - 事件对齐与融合网络

Frame-Event Alignment and Fusion Network for High Frame Rate Tracking

Multi-modal tracking gains attention due to its ability to be more accurate
and robust in complex scenarios compared to traditional RGB-based tracking. Its
key lies in how to fuse multi-modal data and reduce the gap between modalities.
However, multi-modal tracking still severely suffers from data deficiency, thus
resulting in the insufficient learning of fusion modules. Instead of building
such a fusion module, in this paper, we provide a new perspective on
multi-modal tracking by attaching importance to the multi-modal visual prompts.
We design a novel multi-modal prompt tracker (ProTrack), which can transfer the
multi-modal inputs to a single modality by the prompt paradigm. By best
employing the tracking ability of pre-trained RGB trackers learning at scale,
our ProTrack can achieve high-performance multi-modal tracking by only altering
the inputs, even without any extra training on multi-modal data. Extensive
experiments on 5 benchmark datasets demonstrate the effectiveness of the
proposed ProTrack.

本研究提出一种基于 prompt 范式的多模态 prompt 跟踪器 ProTrack，通过最大程度地利用 RGB 跟踪器预训练的跟踪能力，仅通过修改输入即可实现高性能的多模态跟踪，对 5 个基准数据集进行的大量实验证明了 ProTrack 的有效性。