Designing computationally efficient exploration strategies for on-policy first-order methods that attain optimal $\mathcal{O}(1/\epsilon^2)$ sample complexity remains open for solving Markov decision processes (MDP). This manuscript provides an answer to this question from a perspectiv