In the rapidly evolving landscape of spoken question-answering (SQA), the
integration of large language models (LLMs) has emerged as a transformative
development. Conventional approaches often entail the use of separate models
for question audio transcription and answer selection, resulting in significant
resource utilization and error accumulation. To tackle these challenges, we
explore the effectiveness of end-to-end (E2E) methodologies for SQA in the
medical domain. Our study introduces a novel zero-shot SQA approach, compared
to traditional cascade systems. Through a comprehensive evaluation conducted on
a new open benchmark of 8 medical tasks and 48 hours of synthetic audio, we
demonstrate that our approach requires up to 14.7 times fewer resources than a
combined 1.3B parameters LLM with a 1.55B parameters ASR model while improving
average accuracy by 0.5\%. These findings underscore the potential of E2E
methodologies for SQA in resource-constrained contexts.

通过在医学领域中探索端到端方法，我们引入了一种新颖的零 - shot 问答方法，与传统的级联系统相比，我们的方法在 8 个医学任务和 48 小时合成音频的全面评估中，相对于一个带有 1.3B 参数 LLM 和 1.55B 参数 ASR 模型的组合模型，节约资源高达 14.7 倍，同时提高了 0.5% 的平均准确性，凸显了端到端方法在资源受限环境中进行口述问答的潜力。

医学领域零射击端到端口语问答

Zero-Shot End-To-End Spoken Question Answering In Medical Domain

As autonomous driving technology matures, end-to-end methodologies have
emerged as a leading strategy, promising seamless integration from perception
to control via deep learning. However, existing systems grapple with challenges
such as unexpected open set environments and the complexity of black-box
models. At the same time, the evolution of deep learning introduces larger,
multimodal foundational models, offering multi-modal visual and textual
understanding. In this paper, we harness these multimodal foundation models to
enhance the robustness and adaptability of autonomous driving systems, enabling
out-of-distribution, end-to-end, multimodal, and more explainable autonomy.
Specifically, we present an approach to apply end-to-end open-set (any
environment/scene) autonomous driving that is capable of providing driving
decisions from representations queryable by image and text. To do so, we
introduce a method to extract nuanced spatial (pixel/patch-aligned) features
from transformers to enable the encapsulation of both spatial and semantic
features. Our approach (i) demonstrates unparalleled results in diverse tests
while achieving significantly greater robustness in out-of-distribution
situations, and (ii) allows the incorporation of latent space simulation (via
text) for improved training (data augmentation via text) and policy debugging.
We encourage the reader to check our explainer video at
this https URL&feature=youtu.be and to view the
code and demos on our project webpage at this https URL

通过使用多模态基础模型，我们提出了一种方法来增强自主驾驶系统的鲁棒性和适应性，实现了开放环境中的端到端多模态自主驾驶，并能从图像和文本的表示中提供驾驶决策。