A Constructive Prediction of the Generalization Error Across Scales

📖 Description

Published as a conference paper at ICLR 2020 A C ONSTRUCTIVE PREDICTION OF THE GENERALIZATION ERROR ACROSS SCALES Jonathan S. Rosenfeld1Amir Rosenfeld2Yonatan Belinkov13Nir Shavit145 fjonsr,belinkov,shanir [email removed] [email removed] 1Massachusetts Institute of Technology2York University3Harvard University 4Neural Magic Inc5Tel Aviv University ABSTRACT The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted o...

📊 Game Impacts

💭 Reactions

🔗 Sources

• [1] https://arxiv.org/abs/1909.12673