We propose Bayesian Deep Q-Network (BDQN), a practical Thompson sampling based reinforcement learning (RL) Algorithm. Thompson sampling allows for targeted exploration in high dimensions through posterior sampling but is usually computationally expensive. We address this limitation by