Genotoxicity of tobacco smoke and tobacco smoke condensate: a review.

D. DeMarini

DOI: https://doi.org/10.1016/J.MRREV.2004.02.001

2004-11-01

Mutation Research

Abstract:

What problem does this paper attempt to address?

The Dopamine Prediction Error: Contributions to Associative Models of Reward Learning

Helen M. Nasser,Donna J. Calu,Geoffrey Schoenbaum,Melissa J. Sharpe

DOI: https://doi.org/10.3389/fpsyg.2017.00244

IF: 3.8

2017-02-22

Frontiers in Psychology

Abstract:Phasic activity of midbrain dopamine neurons is currently thought to encapsulate the prediction-error signal described in Sutton and Barto's (1981) model-free reinforcement learning algorithm. This phasic signal is thought to contain information about the quantitative value of reward, which transfers to the reward-predictive cue after learning. This is argued to endow the reward-predictive cue with the value inherent in the reward, motivating behavior toward cues signaling the presence of reward. Yet theoretical and empirical research has implicated prediction-error signaling in learning that extends far beyond a transfer of quantitative value to a reward-predictive cue. Here, we review the research which demonstrates the complexity of how dopaminergic prediction errors facilitate learning. After briefly discussing the literature demonstrating that phasic dopaminergic signals can act in the manner described by Sutton and Barto (1981), we consider how these signals may also influence attentional processing across multiple attentional systems in distinct brain circuits. Then, we discuss how prediction errors encode and promote the development of context-specific associations between cues and rewards. Finally, we consider recent evidence that shows dopaminergic activity contains information about causal relationships between cues and rewards that reflect information garnered from rich associative models of the world that can be adapted in the absence of direct experience. In discussing this research we hope to support the expansion of how dopaminergic prediction errors are thought to contribute to the learning process beyond the traditional concept of transferring quantitative value.

psychology, multidisciplinary
Subsecond fluctuations in extracellular dopamine encode reward and punishment prediction errors in humans

L. Paul Sands,Angela Jiang,Brittany Liebenow,Emily DiMarco,Adrian W. Laxton,Stephen B. Tatter,P. Read Montague,Kenneth T. Kishida

DOI: https://doi.org/10.1126/sciadv.adi4927

IF: 13.6

2023-12-03

Science Advances

Abstract:In the mammalian brain, midbrain dopamine neuron activity is hypothesized to encode reward prediction errors that promote learning and guide behavior by causing rapid changes in dopamine levels in target brain regions. This hypothesis (and alternatives regarding dopamine's role in punishment-learning) has limited direct evidence in humans. We report intracranial, subsecond measurements of dopamine release in human striatum measured, while volunteers (i.e., patients undergoing deep brain stimulation surgery) performed a probabilistic reward and punishment learning choice task designed to test whether dopamine release encodes only reward prediction errors or whether dopamine release may also encode adaptive punishment learning signals. Results demonstrate that extracellular dopamine levels can encode both reward and punishment prediction errors within distinct time intervals via independent valence-specific pathways in the human brain.

multidisciplinary sciences
Dopamine reward prediction error signal codes the temporal evaluation of a perceptual decision report

Stefania Sarno,Victor de Lafuente,Ranulfo Romo,Néstor Parga

DOI: https://doi.org/10.1073/pnas.1712479114

IF: 11.1

2017-11-13

Proceedings of the National Academy of Sciences

Abstract:Significance How do animals learn to take correct actions based on uncertain observations? Although dopamine neurons can guide learning in conditioning experiments, their role in decision-making tasks is poorly understood. How can they code reward prediction errors and simultaneously exhibit decision-making processes and beliefs about the state of the environment? Using modeling work and analysis of data recorded from monkeys detecting weak stimuli delivered at uncertain times, we propose some answers to these questions. Specifically, we explain how the certainty about the presence of a stimulus is communicated to midbrain dopamine neurons through transient cortical events and why that certainty becomes visible in their response to a relevant task event.
The Medial Prefrontal Cortex Shapes Dopamine Reward Prediction Errors under State Uncertainty

Clara Kwon Starkweather,Samuel J. Gershman,Naoshige Uchida

DOI: https://doi.org/10.1016/j.neuron.2018.03.036

IF: 16.2

2018-05-01

Neuron

Abstract:Animals make predictions based on currently available information. In natural settings, sensory cues may not reveal complete information, requiring the animal to infer the "hidden state" of the environment. The brain structures important in hidden state inference remain unknown. A previous study showed that midbrain dopamine neurons exhibit distinct response patterns depending on whether reward is delivered in 100% (task 1) or 90% of trials (task 2) in a classical conditioning task. Here we found that inactivation of the medial prefrontal cortex (mPFC) affected dopaminergic signaling in task 2, in which the hidden state must be inferred ("will reward come or not?"), but not in task 1, where the state was known with certainty. Computational modeling suggests that the effects of inactivation are best explained by a circuit in which the mPFC conveys inference over hidden states to the dopamine system. VIDEO ABSTRACT.

neurosciences
A causal link between prediction errors, dopamine neurons and learning

Elizabeth E Steinberg,Ronald Keiflin,Josiah R Boivin,Ilana B Witten,Karl Deisseroth,Patricia H Janak

DOI: https://doi.org/10.1038/nn.3413

IF: 25

2013-05-26

Nature Neuroscience

Abstract:Unexpected rewards activate midbrain dopamine neurons, and this response is proposed to support learning by signaling discrepancies between actual and expected outcomes. Here the authors use optogenetic stimulation to demonstrate a causal role for temporally precise dopamine neuron signaling in cue-reward learning.

neurosciences
Neural Representations of Post-Decision Accuracy and Reward Expectation in the Caudate Nucleus and Frontal Eye Field

Yunshu Fan,Takahiro Doi,Joshua I Gold,Long Ding

DOI: https://doi.org/10.1523/JNEUROSCI.0902-23.2023

2024-01-10

Abstract:Performance monitoring that supports ongoing behavioral adjustments is often examined in the context of either choice confidence for perceptual decisions (i.e., "did I get it right?") or reward expectation for reward-based decisions (i.e., "what reward will I receive?"). However, our understanding of how the brain encodes these distinct evaluative signals remains limited because they are easily conflated, particularly in commonly used two-alternative tasks with symmetric rewards for correct choices. Previously we used a motion-discrimination task with asymmetric rewards to identify neural substrates of forming reward-biased perceptual decisions in the caudate nucleus (part of the striatum in the basal ganglia) and the frontal eye field (FEF, in prefrontal cortex). Here we leveraged this task design to partially decouple estimates of accuracy and reward expectation and examine their impacts on subsequent decisions and their representations in those two brain areas. We identified distinguishable representations of these two evaluative signals in individual caudate and FEF neurons, with regional differences in their distribution patterns and time courses. We observed that well-trained monkeys (both sexes) used both evaluative signals, infrequently but consistently, to adjust their subsequent decisions. We found further that these behavioral adjustments had reliable relationships with the neural representations of both evaluative signals in caudate, but not FEF. These results suggest that the cortico-striatal decision network may use diverse evaluative signals to monitor and adjust decision-making behaviors, adding to our understanding of the different roles that the FEF and caudate nucleus play in a diversity of decision-related computations.
Dopamine Modulates Adaptive Prediction Error Coding in the Human Midbrain and Striatum

Kelly M J Diederen,Hisham Ziauddeen,Martin D Vestergaard,Tom Spencer,Wolfram Schultz,Paul C Fletcher

DOI: https://doi.org/10.1523/JNEUROSCI.1979-16.2016

2017-02-15

Abstract:Learning to optimally predict rewards requires agents to account for fluctuations in reward value. Recent work suggests that individuals can efficiently learn about variable rewards through adaptation of the learning rate, and coding of prediction errors relative to reward variability. Such adaptive coding has been linked to midbrain dopamine neurons in nonhuman primates, and evidence in support for a similar role of the dopaminergic system in humans is emerging from fMRI data. Here, we sought to investigate the effect of dopaminergic perturbations on adaptive prediction error coding in humans, using a between-subject, placebo-controlled pharmacological fMRI study with a dopaminergic agonist (bromocriptine) and antagonist (sulpiride). Participants performed a previously validated task in which they predicted the magnitude of upcoming rewards drawn from distributions with varying SDs. After each prediction, participants received a reward, yielding trial-by-trial prediction errors. Under placebo, we replicated previous observations of adaptive coding in the midbrain and ventral striatum. Treatment with sulpiride attenuated adaptive coding in both midbrain and ventral striatum, and was associated with a decrease in performance, whereas bromocriptine did not have a significant impact. Although we observed no differential effect of SD on performance between the groups, computational modeling suggested decreased behavioral adaptation in the sulpiride group. These results suggest that normal dopaminergic function is critical for adaptive prediction error coding, a key property of the brain thought to facilitate efficient learning in variable environments. Crucially, these results also offer potential insights for understanding the impact of disrupted dopamine function in mental illness.SIGNIFICANCE STATEMENT To choose optimally, we have to learn what to expect. Humans dampen learning when there is a great deal of variability in reward outcome, and two brain regions that are modulated by the brain chemical dopamine are sensitive to reward variability. Here, we aimed to directly relate dopamine to learning about variable rewards, and the neural encoding of associated teaching signals. We perturbed dopamine in healthy individuals using dopaminergic medication and asked them to predict variable rewards while we made brain scans. Dopamine perturbations impaired learning and the neural encoding of reward variability, thus establishing a direct link between dopamine and adaptation to reward variability. These results aid our understanding of clinical conditions associated with dopaminergic dysfunction, such as psychosis.
Reward modulates the association between sensory noise and brain activity during perceptual decision-making

Christian Baeuchl,Nils Kroemer,Shakoor Pooseh,Johannes Petzold,Sebastian Bitzer,Franka Thurm,Shu-Chen Li,Michael N Smolka,Michael N. Smolka

DOI: https://doi.org/10.1016/j.neuropsychologia.2020.107675

IF: 3.054

2020-12-01

Neuropsychologia

Abstract:<p>Perceptual decisions entail the accumulation of evidence until a decision criterion is reached. The amount of noise in this process is inversely related to the behavioral performance of the decision-maker. Hence, reducing the amount of perceived noise could improve performance in perceptual decisions. In this study, we investigated whether providing monetary reward for correct responses in a perceptual decision-making task would enhance performance based on prior research linking noise reduction to the administration of reward. To this end, thirty-one healthy young adults carried out an incentivized dot tracking task (iDT) during recording of functional magnetic resonance imaging (fMRI). Behavioral responses were fitted to a Bayesian version of the drift-diffusion model that, among other parameters, also includes an estimate of sensory noise. Fifty percent of the trials were incentivized to compare rewarded with unrewarded trials regarding behavior, brain responses and estimates of model parameters. In order to establish a link between the noise parameter and fMRI activity, we correlated percent signal change (PSC) values from nucleus accumbens and caudate nucleus with noise levels in rewarded and unrewarded trials respectively. Although reward did not affect behavioral performance and model parameters, the fMRI analyses showed notable differences in nucleus accumbens, caudate nucleus and rostral anterior cingulate cortex in rewarded relative to unrewarded trials. Furthermore, higher PSC within nucleus accumbens was significantly associated with lower sensory noise levels, which was specific to rewarded trials. This work is consistent with previous findings on reward modulation of brain responses and marks a first step towards elucidating the effects of reward-induced noise suppression during perceptual decision-making.</p>

behavioral sciences,psychology, experimental,neurosciences
Immediate and mid-term outcomes of sirolimus-eluting stent implantation for chronic total occlusions.

L. Ge,I. Iakovou,J. Cosgrave,A. Chieffo,M. Montorfano,I. Michev,F. Airoldi,M. Carlino,G. Melzi,G. Sangiorgi,N. Corvaja,A. Colombo

DOI: https://doi.org/10.1016/J.ACCREVIEW.2005.08.235

IF: 39.3

2005-09-01

European Heart Journal

Abstract:
Beyond Reward Prediction Errors: Human Striatum Updates Rule Values During Learning

Ian Ballard,Eric M Miller,Steven T Piantadosi,Noah D Goodman,Samuel M McClure

DOI: https://doi.org/10.1093/cercor/bhx259

IF: 4.861

2017-10-13

Cerebral Cortex

Abstract:Abstract Humans naturally group the world into coherent categories defined by membership rules. Rules can be learned implicitly by building stimulus-response associations using reinforcement learning or by using explicit reasoning. We tested if the striatum, in which activation reliably scales with reward prediction error, would track prediction errors in a task that required explicit rule generation. Using functional magnetic resonance imaging during a categorization task, we show that striatal responses to feedback scale with a “surprise” signal derived from a Bayesian rule-learning model and are inconsistent with RL prediction error. We also find that striatum and caudal inferior frontal sulcus (cIFS) are involved in updating the likelihood of discriminative rules. We conclude that the striatum, in cooperation with the cIFS, is involved in updating the values assigned to categorization rules when people learn using explicit reasoning.

neurosciences
Dopamine, Inference, and Uncertainty

Samuel J. Gershman

DOI: https://doi.org/10.1162/neco_a_01023

IF: 3.278

2017-12-01

Neural Computation

Abstract:The hypothesis that the phasic dopamine response reports a reward prediction error has become deeply entrenched. However, dopamine neurons exhibit several notable deviations from this hypothesis. A coherent explanation for these deviations can be obtained by analyzing the dopamine response in terms of Bayesian reinforcement learning. The key idea is that prediction errors are modulated by probabilistic beliefs about the relationship between cues and outcomes, updated through Bayesian inference. This account can explain dopamine responses to inferred value in sensory preconditioning, the effects of cue preexposure (latent inhibition), and adaptive coding of prediction errors when rewards vary across orders of magnitude. We further postulate that orbitofrontal cortex transforms the stimulus representation through recurrent dynamics, such that a simple error-driven learning rule operating on the transformed representation can implement the Bayesian reinforcement learning update.

computer science, artificial intelligence,neurosciences
Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework

Brian F Sadacca,Joshua L Jones,Geoffrey Schoenbaum

DOI: https://doi.org/10.7554/elife.13665

IF: 7.7

2016-03-07

eLife

Abstract:Midbrain dopamine neurons have been proposed to signal reward prediction errors as defined in temporal difference (TD) learning algorithms. While these models have been extremely powerful in interpreting dopamine activity, they typically do not use value derived through inference in computing errors. This is important because much real world behavior – and thus many opportunities for error-driven learning – is based on such predictions. Here, we show that error-signaling rat dopamine neurons respond to the inferred, model-based value of cues that have not been paired with reward and do so in the same framework as they track the putative cached value of cues previously paired with reward. This suggests that dopamine neurons access a wider variety of information than contemplated by standard TD models and that, while their firing conforms to predictions of TD models in some cases, they may not be restricted to signaling errors from TD predictions.

biology
Encoding Motivation Prediction Errors in the Human Dopaminergic Reward System

Yinmei Ni,Sidong Wang,Jie Su,Jian Li,Xiaohong Wan

DOI: https://doi.org/10.21203/rs.3.rs-51287/v1

2020-01-01

Abstract:Abstract The dopaminergic reward system encoding the reward PE signals is vital for reinforcement learning (RL). Although this reward PE hypothesis has been extensively validated, it remains considerable debates on the alternative account of motivation. In the current study, we diverted the participants’ motivation from the conditioned stimulus (CS)-associated valences to the CS-elicited actions in a variant Pavlovian conditioning task under appetitive and aversive conditions. We found that the regions in the dopaminergic reward system did not encode such bidirectional reward PE signals, but the PE magnitudes, namely, the motivation PE signals. These neural signals without indicating the directions of learning could not be directly used for model-free RL, but probably for model-based control. Specifically, the ventral striatum during the feedback phase might encode the need of adjusting the learning policy, while the putative substantia nigra pars compacta (SNc) in the midbrain and the putamen during the prediction phase might sustain the intended actions. Meanwhile, the primary motor cortex encoded the salience PE signals for model-free RL. Therefore, our findings demonstrate that the human dopaminergic reward system could encode the motivation PE signals to substantialize model-based control, rather than model-free learning, suggesting that its involvement in RL should be motivation-dependent.
Reward Prediction in Prefrontal Cortex and Striatum

xiaochuan pan,rubin wang,masamichi sakagami

DOI: https://doi.org/10.1007/978-94-017-9548-7_10

2015-01-01

Abstract:The prefrontal cortex (PFC) and striatum have mutual connections through direct and indirect pathways, and both are involved in reward prediction. But it has been suggested that the PFC and striatum may have different mechanisms in reward prediction. To understand the nature of reward process in the two areas, we recorded single-unit activity from the lateral PFC (LPFC) and striatum in monkeys performing a reward inference task. We found that prefrontal neurons could predict the reward value of a stimulus even when the monkeys had not yet learned the stimulus-reward association directly. Striatal neurons, however, could predict the reward only after directly experiencing the stimulus-reward contingency. Our results suggested dissociable functions in reward predictions: the LPFC utilized causal structure of the task or higher-order conditioning in a generative process of reward inference, whereas the striatum applied direct experiences of stimulus-reward associations in the guidance of behavior.
Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward

Kenneth T Kishida,Ignacio Saez,Terry Lohrenz,Mark R Witcher,Adrian W Laxton,Stephen B Tatter,Jason P White,Thomas L Ellis,Paul E M Phillips,P Read Montague

DOI: https://doi.org/10.1073/pnas.1513619112

2016-01-05

Abstract:In the mammalian brain, dopamine is a critical neuromodulator whose actions underlie learning, decision-making, and behavioral control. Degeneration of dopamine neurons causes Parkinson's disease, whereas dysregulation of dopamine signaling is believed to contribute to psychiatric conditions such as schizophrenia, addiction, and depression. Experiments in animal models suggest the hypothesis that dopamine release in human striatum encodes reward prediction errors (RPEs) (the difference between actual and expected outcomes) during ongoing decision-making. Blood oxygen level-dependent (BOLD) imaging experiments in humans support the idea that RPEs are tracked in the striatum; however, BOLD measurements cannot be used to infer the action of any one specific neurotransmitter. We monitored dopamine levels with subsecond temporal resolution in humans (n = 17) with Parkinson's disease while they executed a sequential decision-making task. Participants placed bets and experienced monetary gains or losses. Dopamine fluctuations in the striatum fail to encode RPEs, as anticipated by a large body of work in model organisms. Instead, subsecond dopamine fluctuations encode an integration of RPEs with counterfactual prediction errors, the latter defined by how much better or worse the experienced outcome could have been. How dopamine fluctuations combine the actual and counterfactual is unknown. One possibility is that this process is the normal behavior of reward processing dopamine neurons, which previously had not been tested by experiments in animal models. Alternatively, this superposition of error terms may result from an additional yet-to-be-identified subclass of dopamine neurons.
Dopamine reward prediction error coding

Wolfram Schultz

DOI: https://doi.org/10.31887/dcns.2016.18.1/wschultz

2016-03-31

Dialogues in Clinical Neuroscience

Abstract:Reward prediction errors consist of the differences between received and predicted rewards. They are crucial for basic forms of learning about rewards and make us strive for more rewards-an evolutionary beneficial trait. Most dopamine neurons in the midbrain of humans, monkeys, and rodents signal a reward prediction error; they are activated by more reward than predicted (positive prediction error), remain at baseline activity for fully predicted rewards, and show depressed activity with less reward than predicted (negative prediction error). The dopamine signal increases nonlinearly with reward value and codes formal economic utility. Drugs of addiction generate, hijack, and amplify the dopamine reward signal and induce exaggerated, uncontrolled dopamine effects on neuronal plasticity. The striatum, amygdala, and frontal cortex also show reward prediction error coding, but only in subpopulations of neurons. Thus, the important concept of reward prediction errors is implemented in neuronal hardware.
Midbrain signaling of identity prediction errors depends on orbitofrontal cortex networks

Qingfang Liu,Yao Zhao,Sumedha Attanti,Joel L. Voss,Geoffrey Schoenbaum,Thorsten Kahnt

DOI: https://doi.org/10.1038/s41467-024-45880-1

IF: 16.6

2024-02-26

Nature Communications

Abstract:Outcome-guided behavior requires knowledge about the identity of future rewards. Previous work across species has shown that the dopaminergic midbrain responds to violations in expected reward identity and that the lateral orbitofrontal cortex (OFC) represents reward identity expectations. Here we used network-targeted transcranial magnetic stimulation (TMS) and functional magnetic resonance imaging (fMRI) during a trans-reinforcer reversal learning task to test the hypothesis that outcome expectations in the lateral OFC contribute to the computation of identity prediction errors (iPE) in the midbrain. Network-targeted TMS aiming at lateral OFC reduced the global connectedness of the lateral OFC and impaired reward identity learning in the first block of trials. Critically, TMS disrupted neural representations of expected reward identity in the OFC and modulated iPE responses in the midbrain. These results support the idea that iPE signals in the dopaminergic midbrain are computed based on outcome expectations represented in the lateral OFC.

multidisciplinary sciences
Dopamine Prediction Errors and the Relativity of Value

Masamichi Sakagami,Shingo Tanaka

DOI: https://doi.org/10.1007/978-981-10-0207-6_9

2016-01-01

Abstract:Value, or reward prediction, is thought to be generated by neural circuits among the basal ganglia and midbrain dopamine area. Midbrain dopamine neurons calculate the difference between estimated and actual rewards and send the reward prediction error to the basal ganglia, particularly striatum where the value is coded. The value reflects the quality and quantity of reward, the timing of reward delivery, etc. In many cases, however, actually experienced values depend on the context in which rewards are given and cannot be explained merely by physicochemical components of reward.
Dopamine firing plays a dual role in coding reward prediction errors and signaling motivation in a working memory task

Stefania Sarno,Manuel Beirán,Joan Falcó-Roget,Gabriel Diaz-deLeon,Román Rossi-Pool,Ranulfo Romo,Néstor Parga

DOI: https://doi.org/10.1073/pnas.2113311119

IF: 11.1

2022-01-06

Proceedings of the National Academy of Sciences

Abstract:Significance Recent studies have confirmed the role of dopamine firing in reward prediction error, even under perceptual uncertainty. However, little is known about dopamine behavior during the use of working memory or its role in motivation to work for reward. Here, we investigated these issues in a discrimination task. Fast dopamine responses reflected a perceptual bias while remaining consistent with the reward prediction error hypothesis. When the bias increased task difficulty, motivation positively correlated with both performance and dopamine activity. In addition, dopamine slowly ramped up in a motivation-dependent way during the working memory period. Characterizing dopamine neurons’ activity during tasks in which motivation influences behavior could importantly advance our knowledge of dopamine roles in effortful control.

Genotoxicity of tobacco smoke and tobacco smoke condensate: a review.

The Dopamine Prediction Error: Contributions to Associative Models of Reward Learning

Subsecond fluctuations in extracellular dopamine encode reward and punishment prediction errors in humans

Dopamine reward prediction error signal codes the temporal evaluation of a perceptual decision report

The Medial Prefrontal Cortex Shapes Dopamine Reward Prediction Errors under State Uncertainty

A causal link between prediction errors, dopamine neurons and learning

Neural Representations of Post-Decision Accuracy and Reward Expectation in the Caudate Nucleus and Frontal Eye Field

Dopamine Modulates Adaptive Prediction Error Coding in the Human Midbrain and Striatum

Reward modulates the association between sensory noise and brain activity during perceptual decision-making

Immediate and mid-term outcomes of sirolimus-eluting stent implantation for chronic total occlusions.

Beyond Reward Prediction Errors: Human Striatum Updates Rule Values During Learning

Dopamine, Inference, and Uncertainty

Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework

Encoding Motivation Prediction Errors in the Human Dopaminergic Reward System

Reward Prediction in Prefrontal Cortex and Striatum

Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward

Dopamine reward prediction error coding

Midbrain signaling of identity prediction errors depends on orbitofrontal cortex networks

Dopamine Prediction Errors and the Relativity of Value

Dopamine firing plays a dual role in coding reward prediction errors and signaling motivation in a working memory task