This report describes our system towards the automatic diagnosis of skin lesions. We aim to incorporate the expert knowledge of dermatologists into the well known framework of convolutional neural networks (CNN), which have shown impressive performance in many visual recognition tasks.