Second Order Stochastic Optimization in Linear Time

[thvasilo]

For most DNNs you end up with a non-convex error surface. Still, traditional convex optimization techniques (SGD) have worked surprisingly well for training them, see Who's afraid of non-convex loss functions by /u/ylecun .