Combining Conjugate Gradient and Momentum for Unconstrained Stochastic Optimization With Applications to Machine Learning

Due to the influence of stochastic gradients, the existing algorithms suffer from slow convergence, noise explosion, and even failure to converge in practice, which motivates us to propose an accelerated algorithm to tackle these issues. Recognizing the potential of gradient, momentum, and conjugate gradient as promising search directions, we propose a 3-D acceleration algorithm, which uses a weighted combination of these three basis. Specifically, in order to analyze the dynamics of the discrete-time algorithm during the update process, we provide a general framework for approximating the discrete-time algorithm in the weak sense by a continuous-time stochastic differential equation. We exploit the continuous-time formulation together with Lyapunov drift optimization to derive novel adaptive step sizes, which effectively improve the performance of the algorithm in stabilizing noise and accelerating convergence. Extensive numerical experiments demonstrate the proposed algorithm's superiority in convergence rate, computation complexity, and noise robustness compared to state-of-the-art baselines.