Measuring the Algorithmic Efficiency of Neural Networks

📖 Description

Measuring the Algorithmic Ef?ciency of Neural NetworksDanny Hernandez?[email removed]Tom B. [email removed]AbstractThree factors drive the advance of AI: algorithmic innovation, data, and the amount ofcompute available for training. Algorithmic progress has traditionally been more dif?cultto quantify than compute and data. In this work, we argue that algorithmic progress hasan aspect that is both straightforward to measure and interesting: reductions over timein the compute needed to reach past capabilities. We show that the number of ?oating-point operations required to train a classi?er to AlexNet-level performance on ImageNethas decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmicef?ciency doubling every 16 months over a period of 7 years. Notably, this outpaces theoriginal Moore?s law rate of improvement in hardware ef?ciency (11x over this period).We observe that hardware and algorithmic ef?ciency gains multiply and can be on a si...

📊 Game Impacts

💭 Reactions

🔗 Sources

• [1] https://arxiv.org/abs/2005.04305