Applications of model-aware reinforcement learning in Bayesian quantum metrology

Federico Belliardo,Fabio Zoratti,Vittorio Giovannetti

2024-03-09

Abstract:An important practical problem in the field of quantum metrology and sensors is to find the optimal sequences of controls for the quantum probe that realize optimal adaptive estimation. In Belliardo et al.,

Quantum Physics

What problem does this paper attempt to address?

Model-aware reinforcement learning for high-performance Bayesian experimental design in quantum metrology

Federico Belliardo,Fabio Zoratti,Florian Marquardt,Vittorio Giovannetti

2024-02-02

Abstract:Quantum sensors offer control flexibility during estimation by allowing manipulation by the experimenter across various parameters. For each sensing platform, pinpointing the optimal controls to enhance the sensor's precision remains a challenging task. While an analytical solution might be out of reach, machine learning offers a promising avenue for many systems of interest, especially given the capabilities of contemporary hardware. We have introduced a versatile procedure capable of optimizing a wide range of problems in quantum metrology, estimation, and hypothesis testing by combining model-aware reinforcement learning (RL) with Bayesian estimation based on particle filtering. To achieve this, we had to address the challenge of incorporating the many non-differentiable steps of the estimation in the training process, such as measurements and the resampling of the particle filter. Model-aware RL is a gradient-based method, where the derivatives of the sensor's precision are obtained through automatic differentiation (AD) in the simulation of the experiment. Our approach is suitable for optimizing both non-adaptive and adaptive strategies, using neural networks or other agents. We provide an implementation of this technique in the form of a Python library called qsensoropt, alongside several pre-made applications for relevant physical platforms, namely NV centers, photonic circuits, and optical cavities. This library will be released soon on PyPI. Leveraging our method, we've achieved results for many examples that surpass the current state-of-the-art in experimental design. In addition to Bayesian estimation, leveraging model-aware RL, it is also possible to find optimal controls for the minimization of the Cramér-Rao bound, based on Fisher information.

Quantum Physics
Application of machine learning to experimental design in quantum mechanics

Federico Belliardo,Fabio Zoratti,Vittorio Giovannetti

DOI: https://doi.org/10.1142/s0219749924500023

2024-02-23

International Journal of Quantum Information

Abstract:International Journal of Quantum Information, Ahead of Print. The recent advances in Machine Learning hold great promises for the field of quantum sensing and metrology. With the help of reinforcement learning, we can tame the complexity of quantum systems and solve the problem of optimal experimental design. Reinforcement learning is a powerful model-free technique that allows an agent, which is typically a neural network, to learn the best strategy to reach a certain goal in a completely a priori unknown environment. However, in general, we know something about the quantum system the agent is interacting with, at least that it follows the rules of quantum mechanics. In quantum metrology, we typically have a model for the system and only some parameters of the evolution or of the initial state are unknown. We present here a general Machine Learning technique that can optimize the precision of quantum sensors, and in doing so it exploits the knowledge we have on the system. We have developed a Python package to automate a broad class of optimizations that can be found in the tasks of quantum parameter estimation, quantum metrology and quantum hypothesis testing. What the agent is learning here is an optimal adaptive strategy, that, on the basis of the previous outcomes, decides the next measurements to perform. It works both for Bayesian estimation and for frequentist estimation. The user is required to implement the physics of the system to be studied and state which parameters in the experiment are controllable and which are unknowns. The functions of the library allow then to easily train a neural network agent for optimizing the precision of the sensor, by simulating the experiment. We have explored some applications of this technique to magnetometry on NV centers (both DC and AC), to state discrimination in quantum optics and to phase estimation. So far, we were able to certify better results than the current state-of-the-art controls for many examples. The Machine Learning technique developed here can be applied in all those scenarios where the quantum system is well characterized and relatively simple and small. In these cases, we can squeeze every last bit of information from a quantum sensor by controlling it with a neural network appropriately trained.

physics, particles & fields,quantum science & technology, mathematical,computer science, theory & methods
Automatic re-calibration of quantum devices by reinforcement learning

T. Crosta,L. Rebón,F. Vilariño,J. M. Matera,M. Bilkis

2024-04-17

Abstract:During their operation, due to shifts in environmental conditions, devices undergo various forms of detuning from their optimal settings. Typically, this is addressed through control loops, which monitor variables and the device performance, to maintain settings at their optimal values. Quantum devices are particularly challenging since their functionality relies on precisely tuning their parameters. At the same time, the detailed modeling of the environmental behavior is often computationally unaffordable, while a direct measure of the parameters defining the system state is costly and introduces extra noise in the mechanism. In this study, we investigate the application of reinforcement learning techniques to develop a model-free control loop for continuous recalibration of quantum device parameters. Furthermore, we explore the advantages of incorporating minimal environmental noise models. As an example, the application to numerical simulations of a Kennedy receiver-based long-distance quantum communication protocol is presented.

Quantum Physics,Machine Learning
Critical quantum metrology assisted by real-time feedback control

Raffaele Salvia,Mohammad Mehboudi,Martí Perarnau-Llobet

DOI: https://doi.org/10.1103/PhysRevLett.130.240803

2022-11-15

Abstract:We investigate critical quantum metrology,that is the estimation of parameters in many-body systems close to a quantum critical point, through the lens of Bayesian inference theory. We first derive a no-go result stating that any non-adaptive measurement strategy will fail to exploit quantum critical enhancement (i.e. precision beyond the shot-noise limit) for a sufficiently large number of particles $N$ whenever our prior knowledge is limited. We then consider different adaptive strategies that can overcome this no-go result, and illustrate their performance in the estimation of (i) a magnetic field using a probe of 1D spin Ising chain and (ii) the coupling strength in a Bose-Hubbard square lattice. Our results show that adaptive strategies with real-time feedback control can achieve sub-shot noise scaling even with few measurements and substantial prior uncertainty.

Quantum Physics,Statistical Mechanics
Optimized Bayesian system identification in quantum devices

Thomas M. Stace,Jiayin Chen,Li Li,Viktor S. Perunicic,Andre R. R. Carvalho,Michael Hush,Christophe H. Valahu,Ting Rei Tan,and Michael J. Biercuk

DOI: https://doi.org/10.1103/physrevapplied.21.014012

IF: 4.6

2024-01-10

Physical Review Applied

Abstract:Identifying and calibrating quantitative dynamical models for physical quantum systems is important for a variety of applications. Here we present a closed-loop Bayesian learning algorithm for estimating multiple unknown parameters in a dynamical model, using optimized experimental "probe" controls and measurement. The estimation algorithm is based on a Bayesian particle filter, and is designed to autonomously choose informationally optimized probe experiments with which to compare to model predictions. We demonstrate the performance of the algorithm in both simulated calibration tasks and in an experimental single-qubit ion-trap system. Experimentally, we find that, with 60 times fewer samples, we exceed the precision of conventional calibration methods, delivering an approximately 93 times improvement in efficiency (as quantified by the reduction of measurements and resets required to achieve a target residual uncertainty and multiplied by the increase in accuracy). In simulated and experimental demonstrations, we see that successively longer pulses are selected as the posterior uncertainty iteratively decreases, leading to an exponential improvement in the accuracy of model parameters with the number of experimental queries, and a commensurate increase in the per-query time. https://doi.org/10.1103/PhysRevApplied.21.014012 © 2024 American Physical Society

physics, applied
Non-Markovian Quantum Control via Model Maximum Likelihood Estimation and Reinforcement Learning

Tanmay Neema,Susmit Jha,Tuhin Sahai

2024-02-08

Abstract:Reinforcement Learning (RL) techniques have been increasingly applied in optimizing control systems. However, their application in quantum systems is hampered by the challenge of performing closed-loop control due to the difficulty in measuring these systems. This often leads to reliance on assumed models, introducing model bias, a problem that is exacerbated in open quantum dynamics where Markovian approximations are not valid. To address these challenges, we propose a novel approach that incorporates the non-Markovian nature of the environment into a low-dimensional effective reservoir. By initially employing a series of measurements as a 'dataset', we utilize machine learning techniques to learn the effective quantum dynamics more efficiently than traditional tomographic methods. Our methodology aims to demonstrates that by integrating reinforcement learning with model learning, it is possible to devise control policies and models that can counteract decoherence in a spin-boson system. This approach may not only mitigates the issues of model bias but also provides a more accurate representation of quantum dynamics, paving the way for more effective quantum control strategies.

Quantum Physics
Neural-Network Heuristics for Adaptive Bayesian Quantum Estimation

Lukas J. Fiderer,Jonas Schuff,Daniel Braun

DOI: https://doi.org/10.1103/PRXQuantum.2.020303

2021-04-08

Abstract:Quantum metrology promises unprecedented measurement precision but suffers in practice from the limited availability of resources such as the number of probes, their coherence time, or non-classical quantum states. The adaptive Bayesian approach to parameter estimation allows for an efficient use of resources thanks to adaptive experiment design. For its practical success fast numerical solutions for the Bayesian update and the adaptive experiment design are crucial. Here we show that neural networks can be trained to become fast and strong experiment-design heuristics using a combination of an evolutionary strategy and reinforcement learning. Neural-network heuristics are shown to outperform established heuristics for the technologically important example of frequency estimation of a qubit that suffers from dephasing. Our method of creating neural-network heuristics is very general and complements the well-studied sequential Monte-Carlo method for Bayesian updates to form a complete framework for adaptive Bayesian quantum estimation.

Quantum Physics,Machine Learning
Quantum noise modeling through Reinforcement Learning

Simone Bordoni,Andrea Papaluca,Piergiorgio Buttarini,Alejandro Sopena,Stefano Giagu,Stefano Carrazza

2024-08-03

Abstract:In the current era of quantum computing, robust and efficient tools are essential to bridge the gap between simulations and quantum hardware execution. In this work, we introduce a machine learning approach to characterize the noise impacting a quantum chip and emulate it during simulations. Our algorithm leverages reinforcement learning, offering increased flexibility in reproducing various noise models compared to conventional techniques such as randomized benchmarking or heuristic noise models. The effectiveness of the RL agent has been validated through simulations and testing on real superconducting qubits. Additionally, we provide practical use-case examples for the study of renowned quantum algorithms.

Quantum Physics,Computational Physics
A reinforcement learning approach for quantum state engineering

Jelena Mackeprang,Durga B. Rao Dasari,Jörg Wrachtrup

DOI: https://doi.org/10.1007/s42484-020-00016-8

2020-05-11

Quantum Machine Intelligence

Abstract:Machine learning (ML) has become an attractive tool for solving various problems in different fields of physics, including the quantum domain. Here, we show how classical reinforcement learning (RL) could be used as a tool for quantum state engineering (QSE). We employ a measurement based control for QSE where the action sequences are determined by the choice of the measurement basis and the reward through the fidelity of obtaining the target state. Our analysis clearly displays a learning feature in QSE, for example in preparing arbitrary two-qubit entangled states and delivers successful action sequences that generalise previously found human solutions from exact quantum dynamics. We provide a systematic algorithmic approach for using RL for quantum protocols that deal with a non-trivial continuous state space, and discuss the scaling of these approaches for the preparation of larger entangled (cluster) states.

English Else
Optimal and Variational Multi-Parameter Quantum Metrology and Vector Field Sensing

Raphael Kaubruegger,Athreya Shankar,Denis V. Vasilyev,Peter Zoller

DOI: https://doi.org/10.1103/PRXQuantum.4.020333

2023-02-16

Abstract:We study multi-parameter sensing of 2D and 3D vector fields within the Bayesian framework for $SU(2)$ quantum interferometry. We establish a method to determine the optimal quantum sensor, which establishes the fundamental limit on the precision of simultaneously estimating multiple parameters with an $N$-atom sensor. Keeping current experimental platforms in mind, we present sensors that have limited entanglement capabilities, and yet, significantly outperform sensors that operate without entanglement and approach the optimal quantum sensor in terms of performance. Furthermore, we show how these sensors can be implemented on current programmable quantum sensors with variational quantum circuits by minimizing a metrological cost function. The resulting circuits prepare tailored entangled states and perform measurements in an appropriate entangled basis to realize the best possible quantum sensor given the native entangling resources available on a given sensor platform. Notable examples include a 2D and 3D quantum ``compass'' and a 2D sensor that provides a scalable improvement over unentangled sensors. Our results on optimal and variational multi-parameter quantum metrology are useful for advancing precision measurements in fundamental science and ensuring the stability of quantum computers, which can be achieved through the incorporation of optimal quantum sensors in a quantum feedback loop.

Quantum Physics,Atomic Physics
Model-Free Quantum Control with Reinforcement Learning

V. V. Sivak,A. Eickbusch,H. Liu,B. Royer,I. Tsioutsios,M. H. Devoret,V. V. Sivak,M. H. Devoret

DOI: https://doi.org/10.1103/PhysRevX.12.011059

2022-03-29

Physical Review X

Abstract:Model bias is an inherent limitation of the current dominant approach to optimal quantum control, which relies on a system simulation for optimization of control policies. To overcome this limitation, we propose a circuit-based approach for training a reinforcement learning agent on quantum control tasks in a model-free way. Given a continuously parametrized control circuit, the agent learns its parameters through trial-and-error interaction with the quantum system, using measurement outcomes as the only source of information about the quantum state. Focusing on control of a harmonic oscillator coupled to an ancilla qubit, we show how to reward the learning agent with measurements of experimentally available observables. We train the agent to prepare various nonclassical states via both unitary control and control with adaptive measurement-based quantum feedback, and to execute logical gates on encoded qubits. The agent does not rely on averaging for state tomography or fidelity estimation, and significantly outperforms widely used model-free methods in terms of sample efficiency. Our numerical work is of immediate relevance to superconducting circuits and trapped ions platforms where such training can be implemented in experiment, allowing complete elimination of model bias and the adaptation of quantum control policies to the specific system in which they are deployed. https://doi.org/10.1103/PhysRevX.12.011059 Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI. Published by the American Physical Society

physics, multidisciplinary
Hybrid quantum-classical approach to enhanced quantum metrology

Xiaodong Yang,Xi Chen,Jun Li,Xinhua Peng,Raymond Laflamme

DOI: https://doi.org/10.1038/s41598-020-80070-1

IF: 4.6

2021-01-12

Scientific Reports

Abstract:Abstract Quantum metrology plays a fundamental role in many scientific areas. However, the complexity of engineering entangled probes and the external noise raise technological barriers for realizing the expected precision of the to-be-estimated parameter with given resources. Here, we address this problem by introducing adjustable controls into the encoding process and then utilizing a hybrid quantum-classical approach to automatically optimize the controls online. Our scheme does not require any complex or intractable off-line design, and it can inherently correct certain unitary errors during the learning procedure. We also report the first experimental demonstration of this promising scheme for the task of finding optimal probes for frequency estimation on a nuclear magnetic resonance (NMR) processor. The proposed scheme paves the way to experimentally auto-search optimal protocol for improving the metrology precision.

multidisciplinary sciences
Deep Bayesian Experimental Design for Quantum Many-Body Systems

Leopoldo Sarra,Florian Marquardt

2023-06-26

Abstract:Bayesian experimental design is a technique that allows to efficiently select measurements to characterize a physical system by maximizing the expected information gain. Recent developments in deep neural networks and normalizing flows allow for a more efficient approximation of the posterior and thus the extension of this technique to complex high-dimensional situations. In this paper, we show how this approach holds promise for adaptive measurement strategies to characterize present-day quantum technology platforms. In particular, we focus on arrays of coupled cavities and qubit arrays. Both represent model systems of high relevance for modern applications, like quantum simulations and computing, and both have been realized in platforms where measurement and control can be exploited to characterize and counteract unavoidable disorder. Thus, they represent ideal targets for applications of Bayesian experimental design.

Quantum Physics,Machine Learning
Brain-Inspired Agents for Quantum Reinforcement Learning

Eva Andrés,Manuel Pegalajar Cuéllar,Gabriel Navarro

DOI: https://doi.org/10.3390/math12081230

IF: 2.4

2024-04-20

Mathematics

Abstract:In recent years, advancements in brain science and neuroscience have significantly influenced the field of computer science, particularly in the domain of reinforcement learning (RL). Drawing insights from neurobiology and neuropsychology, researchers have leveraged these findings to develop novel mechanisms for understanding intelligent decision-making processes in the brain. Concurrently, the emergence of quantum computing has opened new frontiers in artificial intelligence, leading to the development of quantum machine learning (QML). This study introduces a novel model that integrates quantum spiking neural networks (QSNN) and quantum long short-term memory (QLSTM) architectures, inspired by the complex workings of the human brain. Specifically designed for reinforcement learning tasks in energy-efficient environments, our approach progresses through two distinct stages mirroring sensory and memory systems. In the initial stage, analogous to the brain's hypothalamus, low-level information is extracted to emulate sensory data processing patterns. Subsequently, resembling the hippocampus, this information is processed at a higher level, capturing and memorizing correlated patterns. We conducted a comparative analysis of our model against existing quantum models, including quantum neural networks (QNNs), QLSTM, QSNN and their classical counterparts, elucidating its unique contributions. Through empirical results, we demonstrated the effectiveness of utilizing quantum models inspired by the brain, which outperform the classical approaches and other quantum models in optimizing energy use case. Specifically, in terms of average, best and worst total reward, test reward, robustness, and learning curve.

mathematics
Parameter Estimation in Quantum Metrology Technique for Time Series Prediction

Vaidik A Sharma,N.Madurai Meenachi,B.Venkatraman

2024-06-12

Abstract:The paper investigates the techniques of quantum computation in metrological predictions, with a particular emphasis on enhancing prediction potential through variational parameter estimation. The applicability of quantum simulations and quantum metrology techniques for modelling complex physical systems and achieving high-resolution measurements are proposed. The impacts of various parameter distributions and learning rates on predictive accuracy are investigated. Modelling the time evolution of physical systems Hamiltonian simulation and the product formula procedure are adopted. The time block method is analyzed in order to reduce simulation errors, while the Schatten-infinite norm is used to evaluate the simulation precision. Methodology requires estimation of optimized parameters by minimizing loss functions and resource needs. For this purpose, the mathematical formulations of Cramer Rao Bound and Fischer Information are indispensable requirements. The impact of learning rates on regulating the loss function for various parameter values. Using parameterized quantum circuits, the article outlines a four-step procedure for extracting information. This method involves the preparation of input states, the evolution of parameterized quantum states, the measurement of outputs, and the estimation of parameters based on multiple measurements. The study analyses variational unitary circuits with optimized parameter estimation for more precise predictions. The findings shed light on the effects of normal parameter distributions and learning rates on attaining the most optimal state and comparison with classical Long Short Term Memory (LSTM) predictions, providing valuable insights for the development of more appropriate approaches in quantum computing.

Quantum Physics,Neural and Evolutionary Computing
Improving robustness of quantum feedback control with reinforcement learning

Manuel Guatto,Gian Antonio Susto,Francesco Ticozzi

2024-01-31

Abstract:Obtaining reliable state preparation protocols is a key step towards practical implementation of many quantum technologies, and one of the main tasks in quantum control. In this work, different reinforcement learning approaches are used to derive a feedback law for state preparation of a desired state in a target system. In particular, we focus on the robustness of the obtained strategies with respect to different types and amount of noise. Comparing the results indicates that the learned controls are more robust to unmodeled perturbations with respect to simple feedback strategy based on optimized population transfer, and that training on simulated nominal model retain the same advantages displayed by controllers trained on real data. The possibility of effective off-line training of robust controllers promises significant advantages towards practical implementation.

Quantum Physics
Benchmarking Bayesian quantum estimation

Valeria Cimini,Emanuele Polino,Mauro Valeri,Nicolò Spagnolo,Fabio Sciarrino

2024-01-26

Abstract:The quest for precision in parameter estimation is a fundamental task in different scientific areas. The relevance of this problem thus provided the motivation to develop methods for the application of quantum resources to estimation protocols. Within this context, Bayesian estimation offers a complete framework for optimal quantum metrology techniques, such as adaptive protocols. However, the use of the Bayesian approach requires extensive computational resources, especially in the multiparameter estimations that represent the typical operational scenario for quantum sensors. Hence, the requirement to characterize protocols implementing Bayesian estimations can become a significant challenge. This work focuses on the crucial task of robustly benchmarking the performances of these protocols in both single and multiple-parameter scenarios. By comparing different figures of merits, evidence is provided in favor of using the median of the quadratic error in the estimations in order to mitigate spurious effects due to the numerical discretization of the parameter space, the presence of limited data, and numerical instabilities. These results, providing a robust and reliable characterization of Bayesian protocols, find natural applications to practical problems within the quantum estimation framework.

Quantum Physics
Demonstration of machine-learning-enhanced Bayesian quantum state estimation

Sanjaya Lohani,Joeseph Lukens,Atiyya Davis,Amirali Khannejad,Sangita Regmi,Daniel Jones,Ryan Thomas Glasser,Thomas A Searles,Brian Kirby

DOI: https://doi.org/10.1088/1367-2630/ace6c8

2023-07-13

New Journal of Physics

Abstract:Machine learning (ML) has found broad applicability in quantum information science in topics as diverse as experimental design, state classification, and even studies on quantum foundations. Here, we experimentally realize an approach for defining custom prior distributions that are automatically tuned using ML for Bayesian quantum state estimation methods that generally better conform to the physical properties of the underlying system than standard fixed prior distributions. Previously, researchers have looked to Bayesian quantum state tomography for advantages like uncertainty quantification, the return of reliable estimates under any measurement condition, and minimal mean-squared error. However, practical challenges related to long computation times and conceptual issues concerning how to incorporate prior knowledge most suitably can overshadow these benefits. Using both simulated and experimental measurement results, we demonstrate that ML-defined prior distributions reduce net convergence times and provide a natural way to incorporate both implicit and explicit information directly into the prior distribution. These results constitute a promising path toward practical implementations of Bayesian quantum state tomography.

physics, multidisciplinary

Applications of model-aware reinforcement learning in Bayesian quantum metrology

Model-aware reinforcement learning for high-performance Bayesian experimental design in quantum metrology

Application of machine learning to experimental design in quantum mechanics

Automatic re-calibration of quantum devices by reinforcement learning

Critical quantum metrology assisted by real-time feedback control

Optimized Bayesian system identification in quantum devices

Non-Markovian Quantum Control via Model Maximum Likelihood Estimation and Reinforcement Learning

Neural-Network Heuristics for Adaptive Bayesian Quantum Estimation

Quantum noise modeling through Reinforcement Learning

A reinforcement learning approach for quantum state engineering

Optimal and Variational Multi-Parameter Quantum Metrology and Vector Field Sensing

Model-Free Quantum Control with Reinforcement Learning

Hybrid quantum-classical approach to enhanced quantum metrology

Deep Bayesian Experimental Design for Quantum Many-Body Systems

Brain-Inspired Agents for Quantum Reinforcement Learning

Parameter Estimation in Quantum Metrology Technique for Time Series Prediction

Improving robustness of quantum feedback control with reinforcement learning

Benchmarking Bayesian quantum estimation

Demonstration of machine-learning-enhanced Bayesian quantum state estimation