Physics Nobel scooped by machine-learning pioneers

Elizabeth Gibney,Davide Castelvecchi

DOI: https://doi.org/10.1038/d41586-024-03213-8

IF: 64.8

2024-10-09

Nature

Abstract:John Hopfield and Geoffrey Hinton pioneered computational methods that enabled the development of neural networks. John Hopfield and Geoffrey Hinton pioneered computational methods that enabled the development of neural networks.

multidisciplinary sciences

What problem does this paper attempt to address?

Nobel honors for John Hopfield, who ushered attractor dynamics into neuroscience

David Kleinfeld

DOI: https://doi.org/10.1016/j.neuron.2024.11.002

IF: 16.2

2024-12-07

Neuron

Abstract:John Hopfield's model on collective computation linked the recall of memories with interactions and dynamics associated with disordered magnetic systems. Insights from Hopfield's work catalyzed formulations that link the dynamics and emergent properties of recurrently connected generic neurons with the functional properties and signaling observed from brain circuits.

neurosciences
Physics award honors pioneers in artificial intelligence

Adrian Cho

DOI: https://doi.org/10.1126/science.adt7205

IF: 56.9

2024-10-11

Science

Abstract:Models based in statistical physics led to today's neural networks that power chatbots and self-driving cars

multidisciplinary sciences
Combining high-performance hardware, cloud computing, and deep learning frameworks to accelerate physical simulations: probing the Hopfield network

Vaibhav Vavilala

DOI: https://doi.org/10.1088/1361-6404/ab7027

2020-03-20

Abstract:The synthesis of high-performance computing (particularly graphics processing units), cloud computing services (like Google Colab), and high-level deep learning frameworks (such as PyTorch) has powered the burgeoning field of artificial intelligence. While these technologies are popular in the computer science discipline, the physics community is less aware of how such innovations, freely available online, can improve research and education. In this tutorial, we take the Hopfield network as an example to show how the confluence of these fields can dramatically accelerate physics-based computer simulations and remove technical barriers in implementing such programs, thereby making physics experimentation and education faster and more accessible. To do so, we introduce the cloud, the GPU, and AI frameworks that can be easily repurposed for physics simulation. We then introduce the Hopfield network and explain how to produce large-scale simulations and visualizations for free in the cloud with very little code (fully self-contained in the text). Finally, we suggest programming exercises throughout the paper, geared towards advanced undergraduate students studying physics, biophysics, or computer science.

Computational Physics,Disordered Systems and Neural Networks
Physics-AI Symbiosis

Bahram Jalali,Achuta Kadambi,Vwani Roychowdhury

DOI: https://doi.org/10.48550/arXiv.2109.05959

2021-09-11

Abstract:The phenomenal success of physics in explaining nature and designing hardware is predicated on efficient computational models. A universal codebook of physical laws defines the computational rules and a physical system is an interacting ensemble governed by these rules. Led by deep neural networks, artificial intelligence (AI) has introduced an alternate end-to-end data-driven computational framework, with astonishing performance gains in image classification and speech recognition and fueling hopes for a novel approach to discovering physics itself. These gains, however, come at the expense of interpretability and also computational efficiency; a trend that is on a collision course with the expected end of semiconductor scaling known as the Moore's Law. With focus on photonic applications, this paper argues how an emerging symbiosis of physics and artificial intelligence can overcome such formidable challenges, thereby not only extending the latter's spectacular rise but also transforming the direction of physical science.

Emerging Technologies,Optics
Deep Learning

Yann LeCun,Geoffrey E. Hinton,Yoshua Bengio

Abstract:Machine-learning technology powers many aspects of modern society: from web searches to content filtering on social networks to recommendations on e-commerce websites, and it is increasingly present in consumer products such as cameras and smartphones. Machine-learning systems are used to identify objects in images, transcribe speech into text, match news items, posts or products with users’ interests, and select relevant results of search. Increasingly, these applications make use of a class of techniques called deep learning. Conventional machine-learning techniques were limited in their ability to process natural data in their raw form. For decades, constructing a pattern-recognition or machine-learning system required careful engineering and considerable domain expertise to design a feature extractor that transformed the raw data (such as the pixel values of an image) into a suitable internal representation or feature vector from which the learning subsystem, often a classifier, could detect or classify patterns in the input. Representation learning is a set of methods that allows a machine to be fed with raw data and to automatically discover the representations needed for detection or classification. Deep-learning methods are representation-learning methods with multiple levels of representation, obtained by composing simple but non-linear modules that each transform the representation at one level (starting with the raw input) into a representation at a higher, slightly more abstract level. With the composition of enough such transformations, very complex functions can be learned. For classification tasks, higher layers of representation amplify aspects of the input that are important for discrimination and suppress irrelevant variations. An image, for example, comes in the form of an array of pixel values, and the learned features in the first layer of representation typically represent the presence or absence of edges at particular orientations and locations in the image. The second layer typically detects motifs by spotting particular arrangements of edges, regardless of small variations in the edge positions. The third layer may assemble motifs into larger combinations that correspond to parts of familiar objects, and subsequent layers would detect objects as combinations of these parts. The key aspect of deep learning is that these layers of features are not designed by human engineers: they are learned from data using a general-purpose learning procedure. Deep learning is making major advances in solving problems that have resisted the best attempts of the artificial intelligence community for many years. It has turned out to be very good at discovering intricate structures in high-dimensional data and is therefore applicable to many domains of science, business and government. In addition to beating records in image recognition and speech recognition, it has beaten other machine-learning techniques at predicting the activity of potential drug molecules, analysing particle accelerator data, reconstructing brain circuits, and predicting the effects of mutations in non-coding DNA on gene expression and disease. Perhaps more surprisingly, deep learning has produced extremely promising results for various tasks in natural language understanding, particularly topic classification, sentiment analysis, question answering and language translation. We think that deep learning will have many more successes in the near future because it requires very little engineering by hand, so it can easily take advantage of increases in the amount of available computation and data. New learning algorithms and architectures that are currently being developed for deep neural networks will only accelerate this progress.

Computer Science
A new frontier for Hopfield networks

Dmitry Krotov

DOI: https://doi.org/10.1038/s42254-023-00595-y

IF: 36.273

2023-05-18

Nature Reviews Physics

Abstract:Over the past few years there has been a resurgence of interest in Hopfield networks of associative memory. Dmitry Krotov discusses recent theoretical advances and their broader impact in the context of energy-based neural architectures.

physics, applied, multidisciplinary
Physics-Informed Neural Networks and Extensions

Maziar Raissi,Paris Perdikaris,Nazanin Ahmadi,George Em Karniadakis

2024-08-30

Abstract:In this paper, we review the new method Physics-Informed Neural Networks (PINNs) that has become the main pillar in scientific machine learning, we present recent practical extensions, and provide a specific example in data-driven discovery of governing differential equations.

Machine Learning,Artificial Intelligence
Nonlinear Schrödinger Network

Yiming Zhou,Callen MacPhee,Tingyi Zhou,Bahram Jalali

2024-07-24

Abstract:Deep neural networks (DNNs) have achieved exceptional performance across various fields by learning complex nonlinear mappings from large-scale datasets. However, they encounter challenges such as high computational costs and limited interpretability. To address these issues, hybrid approaches that integrate physics with AI are gaining interest. This paper introduces a novel physics-based AI model called the "Nonlinear Schrödinger Network", which treats the Nonlinear Schrödinger Equation (NLSE) as a general-purpose trainable model for learning complex patterns including nonlinear mappings and memory effects from data. Existing physics-informed machine learning methods use neural networks to approximate the solutions of partial differential equations (PDEs). In contrast, our approach directly treats the PDE as a trainable model to obtain general nonlinear mappings that would otherwise require neural networks. As a type of physics-AI symbiosis, it offers a more interpretable and parameter-efficient alternative to traditional black-box neural networks, achieving comparable or better accuracy in some time series classification tasks while significantly reducing the number of required parameters. Notably, the trained Nonlinear Schrödinger Network is interpretable, with all parameters having physical meanings as properties of a virtual physical system that transforms the data to a more separable space. This interpretability allows for insight into the underlying dynamics of the data transformation process. Applications to time series forecasting have also been explored. While our current implementation utilizes the NLSE, the proposed method of using physics equations as trainable models to learn nonlinear mappings from data is not limited to the NLSE and may be extended to other master equations of physics.

Machine Learning
Computation of the Time-Dependent Dirac Equation with Physics-Informed Neural Networks

Emmanuel Lorin,Xu Yang

DOI: https://doi.org/10.48550/arXiv.2204.02959

2022-04-06

Computational Physics

Abstract:We propose to compute the time-dependent Dirac equation using physics-informed neural networks (PINNs), a new powerful tool in scientific machine learning avoiding the use of approximate derivatives of differential operators. PINNs search solutions in the form of parameterized (deep) neural networks, whose derivatives (in time and space) are performed by automatic differentiation. The computational cost comes from the need to solve high-dimensional optimization problems using stochastic gradient methods and train the network with a large number of points. Specifically, we derive PINNs-based algorithms and present some key fundamental properties of these algorithms when applied to the Dirac equations in different physical frameworks.
Applications of Deep Learning to physics workflows

Manan Agarwal,Jay Alameda,Jeroen Audenaert,Will Benoit,Damon Beveridge,Meghna Bhattacharya,Chayan Chatterjee,Deep Chatterjee,Andy Chen,Muhammed Saleem Cholayil,Chia-Jui Chou,Sunil Choudhary,Michael Coughlin,Maximilian Dax,Aman Desai,Andrea Di Luca,Javier Mauricio Duarte,Steven Farrell,Yongbin Feng,Pooyan Goodarzi,Ekaterina Govorkova,Matthew Graham,Jonathan Guiang,Alec Gunny,Weichangfeng Guo,Janina Hakenmueller,Ben Hawks,Shih-Chieh Hsu,Pratik Jawahar,Xiangyang Ju,Erik Katsavounidis,Manolis Kellis,Elham E Khoda,Fatima Zahra Lahbabi,Van Tha Bik Lian,Mia Liu,Konstantin Malanchev,Ethan Marx,William Patrick McCormack,Alistair McLeod,Geoffrey Mo,Eric Anton Moreno,Daniel Muthukrishna,Gautham Narayan,Andrew Naylor,Mark Neubauer,Michael Norman,Rafia Omer,Kevin Pedro,Joshua Peterson,Michael Pürrer,Ryan Raikman,Shivam Raj,George Ricker,Jared Robbins,Batool Safarzadeh Samani,Kate Scholberg,Alex Schuy,Vasileios Skliris,Siddharth Soni,Niharika Sravan,Patrick Sutton,Victoria Ashley Villar,Xiwei Wang,Linqing Wen,Frank Wuerthwein,Tingjun Yang,Shu-Wei Yeh

2023-06-14

Abstract:Modern large-scale physics experiments create datasets with sizes and streaming rates that can exceed those from industry leaders such as Google Cloud and Netflix. Fully processing these datasets requires both sufficient compute power and efficient workflows. Recent advances in Machine Learning (ML) and Artificial Intelligence (AI) can either improve or replace existing domain-specific algorithms to increase workflow efficiency. Not only can these algorithms improve the physics performance of current algorithms, but they can often be executed more quickly, especially when run on coprocessors such as GPUs or FPGAs. In the winter of 2023, MIT hosted the Accelerating Physics with ML at MIT workshop, which brought together researchers from gravitational-wave physics, multi-messenger astrophysics, and particle physics to discuss and share current efforts to integrate ML tools into their workflows. The following white paper highlights examples of algorithms and computing frameworks discussed during this workshop and summarizes the expected computing needs for the immediate future of the involved fields.

High Energy Physics - Experiment,High Energy Astrophysical Phenomena,General Relativity and Quantum Cosmology
Online Training of Hopfield Networks using Predictive Coding

Ehsan Ganjidoost,Mallory Snow,Jeff Orchard

2024-06-21

Abstract:Neuroscience and Artificial Intelligence (AI) have progressed in tandem, each contributing to our understanding of the brain, and inspiring recent developments in biologically-plausible neural networks (NNs) and learning rules. Predictive coding (PC), and its learning rule, have been shown to approximate error backpropagation in a biologically relevant manner, with local weight updates that depend only on the activity of the pre- and post-synaptic neurons. Unlike traditional feedforward NNs where the flow of information goes in one direction, PC models mimic the brain more accurately by passing information bidirectionally: prediction in one direction, and correction/error in the other. PC models learn by clamping some neurons to target values and running the network to equilibrium. At equilibrium, the network calculates its own error gradients right at the location where they are used for weight updates. Traditional backprop requires the computation graph to be feedforward. However, the PC version of backprop does not have this requirement. Amazingly, no one has demonstrated the application of PC learning directly to recurrent neural networks (RNNs). Hopfield networks (HNs) are RNNs that implement a content-addressable memory, learning patterns (or ``memories'') that can be retrieved from partial or corrupted patterns. In this paper, we show that a HN can be trained using the PC learning rules without modification. To our knowledge, this is the first time PC learning has been applied directly to train a RNN, without the need to unroll it in time. Our results indicate that the PC-trained HNs behave like classical HNs.

Neural and Evolutionary Computing
Advancing Physics Data Analysis through Machine Learning and Physics-Informed Neural Networks

Vasileios Vatellis

2024-10-18

Abstract:In an era increasingly focused on green computing and explainable AI, revisiting traditional approaches in theoretical and phenomenological particle physics is paramount. This project evaluates various machine learning (ML) algorithms-including Nearest Neighbors, Decision Trees, Random Forest, AdaBoost, Naive Bayes, Quadratic Discriminant Analysis (QDA), and XGBoost-alongside standard neural networks and a novel Physics-Informed Neural Network (PINN) for physics data analysis. We apply these techniques to a binary classification task that distinguishes the experimental viability of simulated scenarios based on Higgs observables and essential parameters. Through this comprehensive analysis, we aim to showcase the capabilities and computational efficiency of each model in binary classification tasks, thereby contributing to the ongoing discourse on integrating ML and Deep Neural Networks (DNNs) into physics research. In this study, XGBoost emerged as the preferred choice among the evaluated machine learning algorithms for its speed and effectiveness, especially in the initial stages of computation with limited datasets. However, while standard Neural Networks and Physics-Informed Neural Networks (PINNs) demonstrated superior performance in terms of accuracy and adherence to physical laws, they require more computational time. These findings underscore the trade-offs between computational efficiency and model sophistication.

High Energy Physics - Phenomenology,Machine Learning
Deep Learning for Cognitive Neuroscience

Katherine R. Storrs,Nikolaus Kriegeskorte

DOI: https://doi.org/10.48550/arXiv.1903.01458

2019-03-04

Neurons and Cognition

Abstract:Neural network models can now recognise images, understand text, translate languages, and play many human games at human or superhuman levels. These systems are highly abstracted, but are inspired by biological brains and use only biologically plausible computations. In the coming years, neural networks are likely to become less reliant on learning from massive labelled datasets, and more robust and generalisable in their task performance. From their successes and failures, we can learn about the computational requirements of the different tasks at which brains excel. Deep learning also provides the tools for testing cognitive theories. In order to test a theory, we need to realise the proposed information-processing system at scale, so as to be able to assess its feasibility and emergent behaviours. Deep learning allows us to scale up from principles and circuit models to end-to-end trainable models capable of performing complex tasks. There are many levels at which cognitive neuroscientists can use deep learning in their work, from inspiring theories to serving as full computational models. Ongoing advances in deep learning bring us closer to understanding how cognition and perception may be implemented in the brain -- the grand challenge at the core of cognitive neuroscience.
Machine learning in physics and geometry

Yang-Hui He,Elli Heyes,Edward Hirst

DOI: https://doi.org/10.1016/bs.host.2023.06.002

2023-08-09

Handbook of Statistics

Abstract:We survey some recent applications of machine learning to problems in geometry and theoretical physics. Pure mathematical data has been compiled over the last few decades by the community, and experiments in supervised, semi-supervised, and unsupervised machine learning have found surprising success. We thus advocate the program of machine learning mathematical structures and formulating conjectures via pattern recognition, in other words using artificial intelligence to help one do mathematics.

English Else
AI for Science and Science for AI

Konstantin V. Anokhin,Konstantin S. Novoselov,Stanislav K. Smirnov,Albert R. Efimov,Philipp M. Matveev,,,,,

DOI: https://doi.org/10.21146/0042-8744-2022-3-93-105

2022-01-01

Voprosy filosofii

Abstract:In 2016, Hiroaki Kitano proposed that artificial intelligence (AI) will be able to overcome a number of human cognitive limitations that slow down the process of scientific discovery [Kitano 2016 web]. Since then, the odds of AI being awarded the Nobel Prize have been widely discussed, particularly within academic community [Engineering for Research Symposium web 2020]. At the AI Journey 2021 conference, some renowned representatives of four scientific disciplines (physics, mathematics, neurobiology, philosophy) discussed this issue and then co-authored this article [AI 2021 web]. In the first part of our paper, we critically analyze the role of AI technologies in natural science research: how useful they can be for fundamental science, what the potential of AI in natural and exact sciences is, and what principal limitations it has. Another part of our article discusses a counter-question of what science can do for the future research into AI. Today, it is impossible to imagine machine learning without linear algebra, physics of materials, and brain research. All this falls under what is now commonly referred to as AI, a general umbrella term [Russel, Norvig 2021]. Thus, having served the birth of AI once, how can physics, mathematics, and neuroscience serve it today?
Neural Operator: Is data all you need to model the world? An insight into the impact of Physics Informed Machine Learning

Hrishikesh Viswanath,Md Ashiqur Rahman,Abhijeet Vyas,Andrey Shor,Beatriz Medeiros,Stephanie Hernandez,Suhas Eswarappa Prameela,Aniket Bera

2023-09-18

Abstract:Numerical approximations of partial differential equations (PDEs) are routinely employed to formulate the solution of physics, engineering and mathematical problems involving functions of several variables, such as the propagation of heat or sound, fluid flow, elasticity, electrostatics, electrodynamics, and more. While this has led to solving many complex phenomena, there are some limitations. Conventional approaches such as Finite Element Methods (FEMs) and Finite Differential Methods (FDMs) require considerable time and are computationally expensive. In contrast, data driven machine learning-based methods such as neural networks provide a faster, fairly accurate alternative, and have certain advantages such as discretization invariance and resolution invariance. This article aims to provide a comprehensive insight into how data-driven approaches can complement conventional techniques to solve engineering and physics problems, while also noting some of the major pitfalls of machine learning-based approaches. Furthermore, we highlight, a novel and fast machine learning-based approach (~1000x) to learning the solution operator of a PDE operator learning. We will note how these new computational approaches can bring immense advantages in tackling many problems in fundamental and applied physics.

Artificial Intelligence,Machine Learning,Computational Physics
Neuroevolution of Physics-Informed Neural Nets: Benchmark Problems and Comparative Results

Nicholas Sung Wei Yong,Jian Cheng Wong,Pao-Hsiung Chiu,Abhishek Gupta,Chinchun Ooi,Yew-Soon Ong

DOI: https://doi.org/10.1145/3583133.3596397

2023-12-06

Abstract:The potential of learned models for fundamental scientific research and discovery is drawing increasing attention worldwide. Physics-informed neural networks (PINNs), where the loss function directly embeds governing equations of scientific phenomena, is one of the key techniques at the forefront of recent advances. PINNs are typically trained using stochastic gradient descent methods, akin to their deep learning counterparts. However, analysis in this paper shows that PINNs' unique loss formulations lead to a high degree of complexity and ruggedness that may not be conducive for gradient descent. Unlike in standard deep learning, PINN training requires globally optimum parameter values that satisfy physical laws as closely as possible. Spurious local optimum, indicative of erroneous physics, must be avoided. Hence, neuroevolution algorithms, with their superior global search capacity, may be a better choice for PINNs relative to gradient descent methods. Here, we propose a set of five benchmark problems, with open-source codes, spanning diverse physical phenomena for novel neuroevolution algorithm development. Using this, we compare two neuroevolution algorithms against the commonly used stochastic gradient descent, and our baseline results support the claim that neuroevolution can surpass gradient descent, ensuring better physics compliance in the predicted outputs. %Furthermore, implementing neuroevolution with JAX leads to orders of magnitude speedup relative to standard implementations.

Neural and Evolutionary Computing,Artificial Intelligence,Machine Learning,Computational Physics
Accelerating lattice quantum Monte Carlo simulations using artificial neural networks: Application to the Holstein model

Shaozhi Li,Philip M. Dee,Ehsan Khatami,Steven Johnston

DOI: https://doi.org/10.1103/PhysRevB.100.020302

2019-11-22

Abstract:Monte Carlo (MC) simulations are essential computational approaches with widespread use throughout all areas of science. We present a method for accelerating lattice MC simulations using fully connected and convolutional artificial neural networks that are trained to perform local and global moves i... [Phys. Rev. B 100, 020302(R)] Published Mon Jul 22, 2019

English Else
Physics-informed machine learning

George Em Karniadakis,Ioannis G. Kevrekidis,Lu Lu,Paris Perdikaris,Sifan Wang,Liu Yang

DOI: https://doi.org/10.1038/s42254-021-00314-5

IF: 36.273

2021-05-24

Nature Reviews Physics

Abstract:Despite great progress in simulating multiphysics problems using the numerical discretization of partial differential equations (PDEs), one still cannot seamlessly incorporate noisy data into existing algorithms, mesh generation remains complex, and high-dimensional problems governed by parameterized PDEs cannot be tackled. Moreover, solving inverse problems with hidden physics is often prohibitively expensive and requires different formulations and elaborate computer codes. Machine learning has emerged as a promising alternative, but training deep neural networks requires big data, not always available for scientific problems. Instead, such networks can be trained from additional information obtained by enforcing the physical laws (for example, at random points in the continuous space-time domain). Such physics-informed learning integrates (noisy) data and mathematical models, and implements them through neural networks or other kernel-based regression networks. Moreover, it may be possible to design specialized network architectures that automatically satisfy some of the physical invariants for better accuracy, faster training and improved generalization. Here, we review some of the prevailing trends in embedding physics into machine learning, present some of the current capabilities and limitations and discuss diverse applications of physics-informed learning both for forward and inverse problems, including discovering hidden physics and tackling high-dimensional problems.

physics, applied, multidisciplinary
Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks

Chiara Marullo,Elena Agliari

DOI: https://doi.org/10.3390/e23010034

IF: 2.738

2020-12-29

Entropy

Abstract:The Hopfield model and the Boltzmann machine are among the most popular examples of neural networks. The latter, widely used for classification and feature detection, is able to efficiently learn a generative model from observed data and constitutes the benchmark for statistical learning. The former, designed to mimic the retrieval phase of an artificial associative memory lays in between two paradigmatic statistical mechanics models, namely the Curie-Weiss and the Sherrington-Kirkpatrick, which are recovered as the limiting cases of, respectively, one and many stored memories. Interestingly, the Boltzmann machine and the Hopfield network, if considered to be two cognitive processes (learning and information retrieval), are nothing more than two sides of the same coin. In fact, it is possible to exactly map the one into the other. We will inspect such an equivalence retracing the most representative steps of the research in this field.

physics, multidisciplinary

Physics Nobel scooped by machine-learning pioneers

Nobel honors for John Hopfield, who ushered attractor dynamics into neuroscience

Physics award honors pioneers in artificial intelligence

Combining high-performance hardware, cloud computing, and deep learning frameworks to accelerate physical simulations: probing the Hopfield network

Physics-AI Symbiosis

Deep Learning

A new frontier for Hopfield networks

Physics-Informed Neural Networks and Extensions

Nonlinear Schrödinger Network

Computation of the Time-Dependent Dirac Equation with Physics-Informed Neural Networks

Applications of Deep Learning to physics workflows

Online Training of Hopfield Networks using Predictive Coding

Advancing Physics Data Analysis through Machine Learning and Physics-Informed Neural Networks

Deep Learning for Cognitive Neuroscience

Machine learning in physics and geometry

AI for Science and Science for AI

Neural Operator: Is data all you need to model the world? An insight into the impact of Physics Informed Machine Learning

Neuroevolution of Physics-Informed Neural Nets: Benchmark Problems and Comparative Results

Accelerating lattice quantum Monte Carlo simulations using artificial neural networks: Application to the Holstein model

Physics-informed machine learning

Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks