We depend on our own memory to encode, store, and retrieve our experiences.
However, memory lapses can occur. One promising avenue for achieving memory
augmentation is through the use of augmented reality head-mounted displays to
capture and preserve egocentric videos, a practice commonly referred to as life
logging. However, a significant challenge arises from the sheer volume of video
data generated through life logging, as the current technology lacks the
capability to encode and store such large amounts of data efficiently. Further,
retrieving specific information from extensive video archives requires
substantial computational power, further complicating the task of quickly
accessing desired content. To address these challenges, we propose a memory
augmentation system that involves leveraging natural language encoding for
video data and storing them in a vector database. This approach harnesses the
power of large vision language models to perform the language encoding process.
Additionally, we propose using large language models to facilitate natural
language querying. Our system underwent extensive evaluation using the QA-Ego4D
dataset and achieved state-of-the-art results with a BLEU score of 8.3,
outperforming conventional machine learning models that scored between 3.4 and
5.8. Additionally, in a user study, our system received a higher mean response
score of 4.13/5 compared to the human participants' score of 2.46/5 on
real-life episodic memory tasks.

我们提出了一种记忆增强系统，利用自然语言编码和存储视频数据，并借助大型语言模型进行查询，旨在解决生活记录大量数据和高效检索特定信息的挑战。通过使用 QA-Ego4D 数据集评估我们的系统，在 BLEU 分数上取得了 8.3 的最先进结果，优于传统机器学习模型的 3.4 到 5.8 的成绩。在用户研究中，与人工参与者的 2.46/5 的得分相比，我们的系统在现实生活情景记忆任务上获得了更高的平均回应得分 4.13/5。