Who is afraid of big bad minima? Analysis of gradient-flow in a spiked matrix-tensor model - CEA - Commissariat à l’énergie atomique et aux énergies alternatives
Proceedings/Recueil Des Communications Advances in Neural Information Processing Systems Année : 2019

Who is afraid of big bad minima? Analysis of gradient-flow in a spiked matrix-tensor model

Résumé

Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in optimising high-dimensional non-convex functions and why they find good minima instead of being trapped in spurious ones. Here we present a quantitative theory explaining this behaviour in a spiked matrix-tensor model. Our framework is based on the Kac-Rice analysis of stationary points and a closed-form analysis of gradient-flow originating from statistical physics. We show that there is a well defined region of parameters where the gradient-flow algorithm finds a good global minimum despite the presence of exponentially many spurious local minima. We show that this is achieved by surfing on saddles that have strong negative direction towards the global minima, a phenomenon that is connected to a BBP-type threshold in the Hessian describing the critical points of the landscapes.
Fichier principal
Vignette du fichier
1907.08226.pdf (1.49 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

cea-02529145 , version 1 (02-04-2020)

Identifiants

Citer

Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Lenka Zdeborová. Who is afraid of big bad minima? Analysis of gradient-flow in a spiked matrix-tensor model. Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, Advances in Neural Information Processing Systems, 32, pp.8676-8686, 2019, Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 9781713807933. ⟨cea-02529145⟩
72 Consultations
120 Téléchargements

Altmetric

Partager

More