Large language models, particularly GPT-3, are able to produce high quality
summaries of general domain news articles in few- and zero-shot settings.
However, it is unclear if such models are similarly capable in more
specialized, high-stakes domains such as biomedicine. In this paper, we enlist
domain experts (individuals with medical training) to evaluate summaries of
biomedical articles generated by GPT-3, given zero supervision. We consider
both single- and multi-document settings. In the former, GPT-3 is tasked with
generating regular and plain-language summaries of articles describing
randomized controlled trials; in the latter, we assess the degree to which
GPT-3 is able to \emph{synthesize} evidence reported across a collection of
articles. We design an annotation scheme for evaluating model outputs, with an
emphasis on assessing the factual accuracy of generated summaries. We find that
while GPT-3 is able to summarize and simplify single biomedical articles
faithfully, it struggles to provide accurate aggregations of findings over
multiple documents. We release all data and annotations used in this work.

本文使用专业医学人士对 GPT-3 生成的医学文章摘要进行了评估，发现 GPT-3 虽然能够忠实地总结和简化单篇生物医学文章，但在多篇文章发现的证据综合方面存在困难。

使用 GPT-3 对医学证据进行总结、简化和综合（成功率不同）

Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3  (with Varying Success)

Purpose: Basic surgical skills of suturing and knot tying are an essential
part of medical training. Having an automated system for surgical skills
assessment could help save experts time and improve training efficiency. There
have been some recent attempts at automated surgical skills assessment using
either video analysis or acceleration data. In this paper, we present a novel
approach for automated assessment of OSATS based surgical skills and provide an
analysis of different features on multi-modal data (video and accelerometer
data). Methods: We conduct the largest study, to the best of our knowledge, for
basic surgical skills assessment on a dataset that contained video and
accelerometer data for suturing and knot-tying tasks. We introduce "entropy
based" features - Approximate Entropy (ApEn) and Cross-Approximate Entropy
(XApEn), which quantify the amount of predictability and regularity of
fluctuations in time-series data. The proposed features are compared to
existing methods of Sequential Motion Texture (SMT), Discrete Cosine Transform
(DCT) and Discrete Fourier Transform (DFT), for surgical skills assessment.
Results: We report average performance of different features across all
applicable OSATS criteria for suturing and knot tying tasks. Our analysis shows
that the proposed entropy based features out-perform previous state-of-the-art
methods using video data. For accelerometer data, our method performs better
for suturing only. We also show that fusion of video and acceleration features
can improve overall performance with the proposed entropy features achieving
highest accuracy. Conclusions: Automated surgical skills assessment can be
achieved with high accuracy using the proposed entropy features. Such a system
can significantly improve the efficiency of surgical training in medical
schools and teaching hospitals.

本文介绍了一种基于熵的特征来自动评估 OSATS 基本手术技能的方法，并分析了多模数据（视频和加速度计数据）上不同的特征。结果显示，所提出的熵特征优于现有的视频数据分析的方法，而加速度计数据则仅在缝合中的恢复表现优异，融合视频和加速度数据可以提高总体性能。