Medical document understanding has gained much attention recently. One
representative task is the International Classification of Disease (ICD)
diagnosis code assignment. Existing work adopts either RNN or CNN as the
backbone network because the vanilla BERT cannot handle well long documents
(>2000 to kens). One issue shared across all these approaches is that they are
over specific to the ICD code assignment task, losing generality to give the
whole document-level and sentence-level embedding. As a result, it is not
straight-forward to direct them to other downstream NLU tasks. Motivated by
these observations, we propose Medical Document BERT (MDBERT) for long medical
document understanding tasks. MDBERT is not only effective in learning
representations at different levels of semantics but efficient in encoding long
documents by leveraging a bottom-up hierarchical architecture. Compared to
vanilla BERT solutions: 1, MDBERT boosts the performance up to relatively 20%
on the MIMIC-III dataset, making it comparable to current SOTA solutions; 2, it
cuts the computational complexity on self-attention modules to less than 1/100.
Other than the ICD code assignment, we conduct a variety of other NLU tasks on
a large commercial dataset named as TrialTrove, to showcase MDBERT's strength
in delivering different levels of semantics.

提出了一种名为 MDBERT 的新模型，采用自下而上的分层架构，能够对长且具有多层语义的医学文件进行编码，并在多个 NLU 任务中得到有效应用。