Abstract:Recent advances in scanning transmission electron and scanning probe microscopies have opened exciting opportunities in probing the materials structural parameters and various functional properties in real space with angstrom-level precision. This progress has been accompanied by an exponential increase in the size and quality of datasets produced by microscopic and spectroscopic experimental techniques. These developments necessitate adequate methods for extracting relevant physical and chemical information from the large datasets, for which a priori information on the structures of various atomic configurations and lattice defects is limited or absent. Here we demonstrate an application of deep neural networks to extract information from atomically resolved images including location of the atomic species and type of defects. We develop a 'weakly-supervised' approach that uses information on the coordinates of all atomic species in the image, extracted via a deep neural network, to identify a rich variety of defects that are not part of an initial training set. We further apply our approach to interpret complex atomic and defect transformation, including switching between different coordination of silicon dopants in graphene as a function of time, formation of peculiar silicon dimer with mixed 3-fold and 4-fold coordination, and the motion of molecular 'rotor'. This deep learning based approach resembles logic of a human operator, but can be scaled leading to significant shift in the way of extracting and analyzing information from raw experimental data.

Deep Learning for Automated Experimentation in Scanning Transmission Electron Microscopy

Human-in-the-loop: The future of Machine Learning in Automated Electron Microscopy

Deep learning for electron and scanning probe microscopy: From materials design to atomic fabrication

Advancing electron microscopy using deep learning

Roadmap on Deep Learning for Microscopy

Machine Learning-Enabled Image Classification for Automated Electron Microscopy

Ensemble learning and iterative training (ELIT) machine learning: applications towards uncertainty quantification and automated experiment in atom-resolved microscopy

Deep Learning Analysis on Microscopic Imaging in Materials Science

Review: Deep Learning in Electron Microscopy

The Rise of Data-Driven Microscopy powered by Machine Learning

Post-Experiment Forensics and Human-in-the-Loop Interventions in Explainable Autonomous Scanning Probe Microscopy

Machine Learning for Microscopy Data Analysis: Toward Real-time Optical and Electrical Characterization of Sub-micron Materials

The rise of data‐driven microscopy powered by machine learning

Scientific Exploration with Expert Knowledge (SEEK) in Autonomous Scanning Probe Microscopy with Active Learning

Realizing smart scanning transmission electron microscopy using high performance computing

Machine learning in microscopy – insights, opportunities and challenges

Machine learning enabling high-throughput and remote operations at large-scale user facilities

An Automated Scanning Transmission Electron Microscope Guided by Sparse Data Analytics

Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations

Machine Learning-Based Reward-Driven Tuning of Scanning Probe Microscopy: Towards Fully Automated Microscopy

Machine Learning for Optical Scanning Probe Nanoscopy