Abstract:This article surveys the landscape of semiconductor materials and devices research for the acceleration of machine learning (ML) algorithms. We observe a disconnect between the semiconductor and device physics and engineering communities, and the digital logic and computer hardware architecture communities. The article first provides an overview of the principles of computational complexity and fundamental physical limits to computing and their relation to physical systems. The article then provides an introduction to ML by presenting three key components of ML systems: representation, evaluation, and optimisation. The article then discusses and provides examples of the application of emerging technologies from the demiconductor and device physics domains as solutions to computational problems, alongside a brief overview of emerging devices for computing applications. The article then reviews the landscape of ML accelerators, comparing fixed-function and reprogrammable digital logic with novel devices such as memristors, resistive memories, magnetic memories, and probabilistic bits. We observe broadly lower performance of ML accelerators based on novel devices and materials when compared to those based on digital complimentary metal-oxide semiconductor (CMOS) technology, particularly in the MNIST optical character recognition task, a common ML benchmark, and also highlight the lack of a trend of progress in approaches based on novel materials and devices. Lastly, the article proposes figures of merit for meaningful evaluation and comparison of different ML implementations in the hope of fostering a dialogue between the materials science, device physics, digital logic, and computer architecture communities by providing a common frame of reference for their work.

Compute Trends Across Three Eras of Machine Learning

Deep Learning in the Era of Edge Computing: Challenges and Opportunities

Trends in Energy Estimates for Computing in AI/Machine Learning Accelerators, Supercomputers, and Compute-Intensive Applications

The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?

Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

Machine learning: Trends, perspectives, and prospects

Beyond Human-Level Accuracy: Computational Challenges in Deep Learning

Navigating Scaling Laws: Compute Optimality in Adaptive Model Training

Machine Learning and Computational Mathematics

Roadmap on Emerging Hardware and Technology for Machine Learning

Algorithmic progress in language models

Deep learning, deep change? Mapping the evolution and geography of a general purpose technology

The Power of Training: How Different Neural Network Setups Influence the Energy Demand

Research trends in deep learning and machine learning for cloud computing security

Cambrian explosion of computing and big data in the post-moore era

I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey

4+3 Phases of Compute-Optimal Neural Scaling Laws

Bridging the Band Gap: What Device Physicists Need to Know About Machine Learning

Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

Compute-in-Memory Technologies and Architectures for Deep Learning Workloads

Machine Learning and Big Scientific Data