Corresponding author}In this paper, we explore a novel framework, EGIInet
(Explicitly Guided Information Interaction Network), a model for View-guided
Point cloud Completion (ViPC) task, which aims to restore a complete point
cloud from a partial one with a single view image. In comparison with previous
methods that relied on the global semantics of input images, EGIInet
efficiently combines the information from two modalities by leveraging the
geometric nature of the completion task. Specifically, we propose an explicitly
guided information interaction strategy supported by modal alignment for point
cloud completion. First, in contrast to previous methods which simply use 2D
and 3D backbones to encode features respectively, we unified the encoding
process to promote modal alignment. Second, we propose a novel explicitly
guided information interaction strategy that could help the network identify
critical information within images, thus achieving better guidance for
completion. Extensive experiments demonstrate the effectiveness of our
framework, and we achieved a new state-of-the-art (+16\% CD over XMFnet) in
benchmark datasets despite using fewer parameters than the previous methods.
The pre-trained model and code and are available at
this https URL

该研究论文介绍了一种新颖的框架 EGIInet（Explicitly Guided Information Interaction Network），用于视图引导的点云完整性恢复（ViPC）任务，通过利用完成任务的几何特性，该框架有效地结合了两种信息模态，并使用显式的引导信息交互策略来实现更好的点云完整性恢复指导。

显式引导的信息交互网络用于跨模态点云补全

Explicitly Guided Information Interaction Network for Cross-modal Point  Cloud Completion

By leveraging the power of Large Language Models(LLMs) and speech foundation
models, state of the art speech-text bimodal works can achieve challenging
tasks like spoken translation(ST) and question answering(SQA) altogether with
much simpler architectures. In this paper, we utilize the capability of Whisper
encoder and pre-trained Yi-6B. Empirical results reveal that modal alignment
can be achieved with one layer module and hundred hours of speech-text
multitask corpus. We further swap the Yi-6B with human preferences aligned
version of Yi-6B-Chat during inference, and discover that the alignment
capability is applicable as well. In addition, the alignment subspace revealed
by singular value decomposition(SVD) also implies linear alignment subspace is
sparse, which leaves the possibility to concatenate other features like
voice-print or video to expand modality.

利用大型语言模型（LLMs）和语音基础模型，先进的语音 - 文本双模工作可以实现复杂的任务，如口语翻译（ST）和问题回答（SQA），同时具有更简单的结构。本文利用 Whisper 编码器和预训练的 Yi-6B 的功能，通过经验结果发现，使用一个层的模块和百小时的语音 - 文本多任务语料库可以实现模态对齐。我们在推理过程中进一步交换 Yi-6B 为与人类偏好相符的 Yi-6B-Chat 版本，发现模态对齐能力同样适用。此外，奇异值分解（SVD）揭示的对齐子空间还意味着线性对齐子空间是稀疏的，这为连接其他特征（如声纹或视频）以扩展模态性留下了可能性。

跨可转移的语音转文大型语言模型对齐模块

Transferable speech-to-text large language model alignment module

In this paper, we focus on solving one of the most important tasks in the
field of speech processing, i.e., automatic speech recognition (ASR), with
speech foundation encoders and large language models (LLM). Recent works have
complex designs such as compressing the output temporally for the speech
encoder, tackling modal alignment for the projector, and utilizing
parameter-efficient fine-tuning for the LLM. We found that delicate designs are
not necessary, while an embarrassingly simple composition of off-the-shelf
speech encoder, LLM, and the only trainable linear projector is competent for
the ASR task. To be more specific, we benchmark and explore various
combinations of LLMs and speech encoders, leading to the optimal LLM-based ASR
system, which we call SLAM-ASR. The proposed SLAM-ASR provides a clean setup
and little task-specific design, where only the linear projector is trained. To
the best of our knowledge, SLAM-ASR achieves the best performance on the
Librispeech benchmark among LLM-based ASR models and even outperforms the
latest LLM-based audio-universal model trained on massive pair data. Finally,
we explore the capability emergence of LLM-based ASR in the process of modal
alignment. We hope that our study can facilitate the research on extending LLM
with cross-modality capacity and shed light on the LLM-based ASR community.

通过对大型语言模型和语音基础编码器进行多种组合的基准测试和研究，本文提出了一种简单而高效的 SLAM-ASR 系统，该系统在 Librispeech 基准测试中表现出色，并且超过了最新的音频通用模型。同时，本文还探讨了基于 LLM 的 ASR 的能力发展和模态对齐的问题。