We introduce an open-source system called SIGMA (short for "Situated
Interactive Guidance, Monitoring, and Assistance") as a platform for conducting
research on task-assistive agents in mixed-reality scenarios. The system
leverages the sensing and rendering affordances of a head-mounted mixed-reality
device in conjunction with large language and vision models to guide users step
by step through procedural tasks. We present the system's core capabilities,
discuss its overall design and implementation, and outline directions for
future research enabled by the system. SIGMA is easily extensible and provides
a useful basis for future research at the intersection of mixed reality and AI.
By open-sourcing an end-to-end implementation, we aim to lower the barrier to
entry, accelerate research in this space, and chart a path towards
community-driven end-to-end evaluation of large language, vision, and
multimodal models in the context of real-world interactive applications.

SIGMA 是一个开源系统，结合大型语言和视觉模型以及头戴式混合现实设备的感知和渲染特性，通过逐步引导用户完成程序化任务，提供在混合现实场景中研究任务辅助代理的平台，为混合现实与人工智能交叉领域的未来研究提供可扩展的基础。

SIGMA：一个开源的混合现实任务辅助研究系统

SIGMA: An Open-Source Interactive System for Mixed-Reality Task  Assistance Research

Multi-modal semantic segmentation significantly enhances AI agents'
perception and scene understanding, especially under adverse conditions like
low-light or overexposed environments. Leveraging additional modalities
(X-modality) like thermal and depth alongside traditional RGB provides
complementary information, enabling more robust and reliable segmentation. In
this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic
segmentation, utilizing the Selective Structured State Space Model, Mamba.
Unlike conventional methods that rely on CNNs, with their limited local
receptive fields, or Vision Transformers (ViTs), which offer global receptive
fields at the cost of quadratic complexity, our model achieves global receptive
fields coverage with linear complexity. By employing a Siamese encoder and
innovating a Mamba fusion mechanism, we effectively select essential
information from different modalities. A decoder is then developed to enhance
the channel-wise modeling ability of the model. Our method, Sigma, is
rigorously evaluated on both RGB-Thermal and RGB-Depth segmentation tasks,
demonstrating its superiority and marking the first successful application of
State Space Models (SSMs) in multi-modal perception tasks. Code is available at
this https URL

该研究介绍了 Sigma，一种用于多模态语义分割的 Siamese Mamba 网络，利用选择性结构状态空间模型（Mamba），并在 RGB - 热力图和 RGB - 深度分割任务上进行了评估，展示了其在多模态感知任务中的优越性和标志性应用

Sigma: 双子曼巴网络用于多模态语义分割

Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation

Resolution in deep convolutional neural networks (CNNs) is typically bounded
by the receptive field size through filter sizes, and subsampling layers or
strided convolutions on feature maps. The optimal resolution may vary
significantly depending on the dataset. Modern CNNs hard-code their resolution
hyper-parameters in the network architecture which makes tuning such
hyper-parameters cumbersome. We propose to do away with hard-coded resolution
hyper-parameters and aim to learn the appropriate resolution from data. We use
scale-space theory to obtain a self-similar parametrization of filters and make
use of the N-Jet: a truncated Taylor series to approximate a filter by a
learned combination of Gaussian derivative filters. The parameter sigma of the
Gaussian basis controls both the amount of detail the filter encodes and the
spatial extent of the filter. Since sigma is a continuous parameter, we can
optimize it with respect to the loss. The proposed N-Jet layer achieves
comparable performance when used in state-of-the art architectures, while
learning the correct resolution in each layer automatically. We evaluate our
N-Jet layer on both classification and segmentation, and we show that learning
sigma is especially beneficial for inputs at multiple sizes.

使用 N-Jet 层，通过学习 Gaussian 导数滤波器的组合来自动学习每层正确的分辨率，优化连续参数 σ 从而实现有效的分类和分割。

利用尺度空间理论进行深度卷积网络的分辨率学习

Resolution learning in deep convolutional networks using scale-space  theory

The Internet has enabled the creation of a growing number of large-scale
knowledge bases in a variety of domains containing complementary information.
Tools for automatically aligning these knowledge bases would make it possible
to unify many sources of structured knowledge and answer complex queries.
However, the efficient alignment of large-scale knowledge bases still poses a
considerable challenge. Here, we present Simple Greedy Matching (SiGMa), a
simple algorithm for aligning knowledge bases with millions of entities and
facts. SiGMa is an iterative propagation algorithm which leverages both the
structural information from the relationship graph as well as flexible
similarity measures between entity properties in a greedy local search, thus
making it scalable. Despite its greedy nature, our experiments indicate that
SiGMa can efficiently match some of the world's largest knowledge bases with
high precision. We provide additional experiments on benchmark datasets which
demonstrate that SiGMa can outperform state-of-the-art approaches both in
accuracy and efficiency.

本文介绍了一个名为 SiGMa 的算法，它是一种对齐具有数百万实体和事实的知识库的简单且高效的算法，并且可以在准确性和效率方面优于现有方法。