High performing deep neural networks come at the cost of computational complexity that limits its practicality for deployment on portable devices. We propose low-rank transformer (LRT), a memory-efficient and fast neura