The field of statistical machine learning has seen a rapid progress in complex hierarchical Bayesian models. In stochastic variational inference (SVI), the inference problem is mapped to an optimization problem involving stochastic gradients. While this scheme was shown to scale up to