As large language models (LLMs) increasingly permeate daily lives, there is a
growing demand for real-time interactions that mirror human conversations.
Traditional turn-based chat systems driven by LLMs prevent users from verbally
interacting with the system while it is generating responses. To overcome these
limitations, we adapt existing LLMs to \textit{duplex models} so that these
LLMs can listen for users while generating output and dynamically adjust
themselves to provide users with instant feedback. % such as in response to
interruptions. Specifically, we divide the queries and responses of
conversations into several time slices and then adopt a
time-division-multiplexing (TDM) encoding-decoding strategy to
pseudo-simultaneously process these slices. Furthermore, to make LLMs
proficient enough to handle real-time conversations, we build a fine-tuning
dataset consisting of alternating time slices of queries and responses as well
as covering typical feedback types in instantaneous interactions. Our
experiments show that although the queries and responses of conversations are
segmented into incomplete slices for processing, LLMs can preserve their
original performance on standard benchmarks with a few fine-tuning steps on our
dataset. Automatic and human evaluation indicate that duplex models make
user-AI interactions more natural and human-like, and greatly improve user
satisfaction compared to vanilla LLMs. Our duplex model and dataset will be
released.

大型语言模型的双工模型可以实现用户和 AI 的实时对话，通过时间分割多路复用和动态调整，提供即时反馈，改进了用户满意度。

超越回合制游戏：利用 Duplex 模型实现实时对话

Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex  Models

Recent advancements in Large Language Models (LLMs) have significantly
enhanced the capabilities of conversational agents, making them applicable to
various fields (e.g., education). Despite their progress, the evaluation of the
agents often overlooks the complexities of real-world conversations, such as
real-time interactions, multi-party dialogues, and extended contextual
dependencies. To bridge this gap, we introduce DialSim, a real-time dialogue
simulator. In this simulator, an agent is assigned the role of a character from
popular TV shows, requiring it to respond to spontaneous questions using past
dialogue information and to distinguish between known and unknown information.
Key features of DialSim include evaluating the agent's ability to respond
within a reasonable time limit, handling long-term multi-party dialogues, and
managing adversarial settings (e.g., swap character names) to challenge the
agent's reliance on pre-trained knowledge. We utilized this simulator to
evaluate the latest conversational agents and analyze their limitations. Our
experiments highlight both the strengths and weaknesses of these agents,
providing valuable insights for future improvements in the field of
conversational AI. DialSim is available at
this https URL

通过引入实时对话模拟器 DialSim，可以对最新的对话代理进行评估并分析它们的局限性，为未来的对话人工智能领域的改进提供有价值的见解。

DialSim：一个用于评估对话代理的长期对话理解能力的实时模拟器

DialSim: A Real-Time Simulator for Evaluating Long-Term Dialogue  Understanding of Conversational Agents

Highly autonomous generative agents powered by large language models promise
to simulate intricate social behaviors in virtual societies. However, achieving
real-time interactions with humans at a low computational cost remains
challenging. Here, we introduce Lyfe Agents. They combine low-cost with
real-time responsiveness, all while remaining intelligent and goal-oriented.
Key innovations include: (1) an option-action framework, reducing the cost of
high-level decisions; (2) asynchronous self-monitoring for better
self-consistency; and (3) a Summarize-and-Forget memory mechanism, prioritizing
critical memory items at a low cost. We evaluate Lyfe Agents' self-motivation
and sociability across several multi-agent scenarios in our custom LyfeGame 3D
virtual environment platform. When equipped with our brain-inspired techniques,
Lyfe Agents can exhibit human-like self-motivated social reasoning. For
example, the agents can solve a crime (a murder mystery) through autonomous
collaboration and information exchange. Meanwhile, our techniques enabled Lyfe
Agents to operate at a computational cost 10-100 times lower than existing
alternatives. Our findings underscore the transformative potential of
autonomous generative agents to enrich human social experiences in virtual
worlds.

利用 Lyfe Agents 实现了高度自治生成智能代理，能够在虚拟社会中模拟复杂的社交行为，并能以低计算成本与人进行实时互动，通过自主协作和信息交流解决犯罪等问题，同时比现有替代方案的计算成本低 10-100 倍。