Conversational AI systems can engage in unsafe behaviour when handling users'
medical queries that can have severe consequences and could even lead to
deaths. Systems therefore need to be capable of both recognising the
seriousness of medical inputs and producing responses with appropriate levels
of risk. We create a corpus of human written English language medical queries
and the responses of different types of systems. We label these with both
crowdsourced and expert annotations. While individual crowdworkers may be
unreliable at grading the seriousness of the prompts, their aggregated labels
tend to agree with professional opinion to a greater extent on identifying the
medical queries and recognising the risk types posed by the responses. Results
of classification experiments suggest that, while these tasks can be automated,
caution should be exercised, as errors can potentially be very serious.

病人咨询聊天机器人可能存在安全风险，研究表明需要在较高水平意识医疗词汇的基础上，对聊天机器人进行编程设置以适应各种不同的风险等级，同时协同众包技术来对医疗问题进行标注，以提高有效性，而自动化技术在此应用上还需要更多的谨慎和实验验证。