In recent years, Large Language Models (LLMs) have demonstrated an impressive
ability to encode knowledge during pre-training on large text corpora. They can
leverage this knowledge for downstream tasks like question answering (QA), even
in complex areas involving health topics. Considering their high potential for
facilitating clinical work in the future, understanding the quality of encoded
medical knowledge and its recall in LLMs is an important step forward. In this
study, we examine the capability of LLMs to exhibit medical knowledge recall by
constructing a novel dataset derived from systematic reviews -- studies
synthesizing evidence-based answers for specific medical questions. Through
experiments on the new MedREQAL dataset, comprising question-answer pairs
extracted from rigorous systematic reviews, we assess six LLMs, such as GPT and
Mixtral, analyzing their classification and generation performance. Our
experimental insights into LLM performance on the novel biomedical QA dataset
reveal the still challenging nature of this task.

近年来，大型语言模型（LLMs）在大文本语料库的预训练过程中展示了令人印象深刻的知识编码能力。本研究通过构建一个新的基于系统评述的数据集，评估了六种 LLMs（如 GPT 和 Mixtral）在医学知识回忆方面的能力，并揭示了此任务的复杂性。

MedREQAL: 通过问答方式检验大型语言模型对医学知识的记忆能力

MedREQAL: Examining Medical Knowledge Recall of Large Language Models  via Question Answering

Educational chatbots come with a promise of interactive and personalized
learning experiences, yet their development has been limited by the restricted
free interaction capabilities of available platforms and the difficulty of
encoding knowledge in a suitable format. Recent advances in language learning
models with zero-shot learning capabilities, such as ChatGPT, suggest a new
possibility for developing educational chatbots using a prompt-based approach.
We present a case study with a simple system that enables mixed-turn chatbot
interactions and we discuss the insights and preliminary guidelines obtained
from initial tests. We examine ChatGPT's ability to pursue multiple
interconnected learning objectives, adapt the educational activity to users'
characteristics, such as culture, age, and level of education, and its ability
to use diverse educational strategies and conversational styles. Although the
results are encouraging, challenges are posed by the limited history maintained
for the conversation and the highly structured form of responses by ChatGPT, as
well as their variability, which can lead to an unexpected switch of the
chatbot's role from a teacher to a therapist. We provide some initial
guidelines to address these issues and to facilitate the development of
effective educational chatbots.

本研究基于 ChatGPT 模型实现了零 - shot 学习的教育聊天机器人，能够以多种不同的教育策略和对话风格进行个性化学习交互，此外还讨论了聊天历史记录和 ChatGPT 响应变异性带来的挑战，并提供了初步的解决方案以促进有效的教育聊天机器人的开发。

使用 ChatGPT 提示开发有效的教育聊天机器人：社交媒体素养案例初步测试的见解

Developing Effective Educational Chatbots with ChatGPT prompts: Insights  from Preliminary Tests in a Case Study on Social Media Literacy

Protein representation learning has primarily benefited from the remarkable
development of language models (LMs). Accordingly, pre-trained protein models
also suffer from a problem in LMs: a lack of factual knowledge. The recent
solution models the relationships between protein and associated knowledge
terms as the knowledge encoding objective. However, it fails to explore the
relationships at a more granular level, i.e., the token level. To mitigate
this, we propose Knowledge-exploited Auto-encoder for Protein (KeAP), which
performs token-level knowledge graph exploration for protein representation
learning. In practice, non-masked amino acids iteratively query the associated
knowledge tokens to extract and integrate helpful information for restoring
masked amino acids via attention. We show that KeAP can consistently outperform
the previous counterpart on 9 representative downstream applications, sometimes
surpassing it by large margins. These results suggest that KeAP provides an
alternative yet effective way to perform knowledge enhanced protein
representation learning.

KeAP 提出了一种利用知识的自编码器方法，通过 token-level 知识图谱探索辅助蛋白质表示学习，取得了比之前更好的结果。