With the success of down streaming task using English pre-trained langueage model, the pre-trained Chinese language model is also necessary to get a better performance of chinese nlp task. Unlike the English language, Chinese has its special characters such as →