Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions - CEA - Commissariat à l’énergie atomique et aux énergies alternatives Accéder directement au contenu
Proceedings/Recueil Des Communications Advances in Neural Information Processing Systems Année : 2019

Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

Résumé

We study the dynamics of optimization and the generalization properties of one-hidden layer neural networks with quadratic activation function in the over-parametrized regime where the layer width $m$ is larger than the input dimension $d$. We consider a teacher-student scenario where the teacher has the same structure as the student with a hidden layer of smaller width $m^*\le m$. We describe how the empirical loss landscape is affected by the number $n$ of data samples and the width $m^*$ of the teacher network. In particular we determine how the probability that there be no spurious minima on the empirical loss depends on $n$, $d$, and $m^*$, thereby establishing conditions under which the neural network can in principle recover the teacher. We also show that under the same conditions gradient descent dynamics on the empirical loss converges and leads to small generalization error, i.e. it enables recovery in practice. Finally we characterize the time-convergence rate of gradient descent in the limit of a large number of samples. These results are confirmed by numerical experiments.

Dates et versions

cea-04539333 , version 1 (09-04-2024)

Identifiants

Citer

Stefano Sarao Mannelli, Eric Vanden-Eijnden. Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions. 34th Conference on Neural Information Processing Systems, NeurIPS 2020, Advances in Neural Information Processing Systems, 2020, pp.Code 169463, 2019, 10495258. ⟨cea-04539333⟩
7 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More