Prediction error for free monetary reward in the human prefrontal cortex
Introduction
The execution of goal-directed behavior is always followed by monitoring for the successful achievement of the goal. For this process to work effectively, a representation of the expected goal must be compared with the actual outcome. The most widely studied types of outcome in nonhuman primates are appetitive rewards (e.g., fruit juice or food pellets) (Hassani et al., 2001, Tremblay and Schultz, 1999). The neural mechanisms of reward-related prediction error have been extensively studied in nonhuman primates (Hollerman et al., 1998, Leon and Shadlen, 1999, Schultz et al., 1992, Tremblay and Schultz, 1999, Tremblay and Schultz, 2000b, Tremblay et al., 1998, Watanabe et al., 2001).
The midbrain dopamine systems of the primate brain send projections to the basal ganglia and widespread regions of the frontal lobes (Ghashghaei and Barbas, 2001, Goldman-Rakic et al., 1989). These routes are important for conveying reward-related information to frontostriatal circuitry involved in cognitive processing. Importantly, the firing characteristics of dopamine neurons are determined by the ability of animals to predict rewards in advance of their occurrence and whether predictions about outcomes are violated or verified. Animals can be trained to expect a reward if it is consistently preceded by an instruction cue. During learning, as the reward becomes increasingly predictable, its ability to elicit activity in dopamine neurons transfers to the conditioned stimulus, and activity time-locked to the reward itself declines (Schultz, 1998, Schultz et al., 1993). The same neurons respond to the nondelivery of expected rewards in trained animals. In this situation, dopamine neurons phasically decrease their activity at the time that the rewards are expected. Activity in midbrain dopamine neurons therefore has three important characteristics. They signal (i) the prediction of a future reward, (ii) its unexpected occurrence (increased activity on reward presentation), (iii) and its unexpected absence (decreased activity at the time that reward was expected). All these are important for driving and maintaining reward-based learning (Schultz, 1997, Suri and Schultz, 1999, Waelti et al., 2001).
Studies in both humans and nonhuman primates have shown that frontostriatal circuits are important for mediating the influence of reward expectation on the selection and preparation of actions. Specific dopamine-rich regions within the prefrontal cortex (Goldman-Rakic et al., 1992, Lidow et al., 1991, Sawaguchi and Goldman-Rakic, 1991), the premotor cortex (Sawaguchi, 1997), and the striatum (Hassani et al., 2001, Hikosaka et al., 1989, Lauwereyns et al., 2002) contain neurons in which task-related activity is altered when the goals of the task are rewards. Activity related to information processing during delays between instruction cues and manual responses can be altered if the cues also signal the level of reward to be expected (Leon and Shadlen, 1999, Ramnani and Miall, 2003). Neuronal activity of this kind is altered by local infusions of dopamine and its antagonists (Sawaguchi et al., 1986, Sawaguchi et al., 1988, Sawaguchi et al., 1990), suggesting that reward influences this activity through the action of dopamine.
During delayed-response studies of this kind, subjects form associations between instruction stimuli and rewards, but the delivery of rewards is contingent on the performance of correct responses and the outcome is only likely to be a reward if a response is executed. The occurrence of prediction error-related activity in frontostriatal circuits might therefore be specific to the context of actions where the goals are rewards. Although the orbitofrontal cortex is well known to have an essential role in motivational influences on behavior (Elliott et al., 2000a, Rolls, 2001, Rolls, 2000, Schultz et al., 2000, Tremblay and Schultz, 1999, Tremblay and Schultz, 2000a), several studies in nonhuman primates have also demonstrated that neurons in the lateral convexity of the prefrontal cortex encode reward expectancy (Hikosaka and Watanabe, 2000, Watanabe, 1990, Watanabe, 1996, Watanabe et al., 2001). Activity in orbital and lateral prefrontal cortex therefore reflects predictive processing for rewards. If these areas encode predictions about impending rewards, they may also have a role in registering the occurrence of reward-related prediction error.
While most studies of the kind described above have examined activity in āoperantā models of behavior (where the contingency between conditioned stimuli and rewarding outcomes depends on behavior), little is known about the circuitry involved in the direct formation associations between predictive conditioned stimuli and rewarding outcomes independently of behavior (āclassicalā contingency between the conditioned stimulus and the outcome). Single neurons in the prefrontal cortex can encode conditioned stimuli and outcomes, but this finding is not sufficient to demonstrate that it encodes associations between these. A more stringent criterion is to determine whether the neuron responds specifically to violations of cueāreward associations. Although the functional anatomy of monetary reward has been investigated with functional neuroimaging methods (Elliott et al., 1997, Elliott et al., 2000b, Knutson et al., 2000, Knutson et al., 2001a, O'Doherty et al., 2001, Ramnani and Miall, 2003, Thut et al., 1997), no study has examined prediction error-related activity in the human brain for classical contingencies between conditioned stimuli and monetary reward. The main aim of the present study was therefore to localize regions of the human brain that in which there were event-related changes in activity specific for two types of reward-related prediction error when monetary rewards were delivered independently of goal-directed actions.
We used event-related fMRI in human subjects to examine activity related the failure of expected rewards, and the occurrence of unexpected rewards, where outcomes were not contingent on any behavior. We compared each type of prediction error with its corresponding control condition in which the predicted event occurred as expected. We demonstrate that these prediction errors evoke activity in two separate frontotemporal networks.
Section snippets
Subjects
Six healthy volunteers were recruited after they had given written informed consent. The study had local ethical approval.
Stimulus presentation
Subjects lay supine in the scanner and were able to clearly view visual stimuli on a mirror positioned above the eyes 45Ā° to the horizontal, onto which stimuli were projected with an SVGA projector. TTL pulses time-locked with stimulus presentation (controlled by Cogent, http://www.fil.ion.ucl.ac.uk, running on a PowerPC) and image acquisition were simultaneously sampled at
Activity common to all conditions: [A B C D]
We assessed the internal validity of our experimental design and methods for modeling the hemodynamic response by testing for predictable effects that were orthogonal to the contrasts of interest. Visual stimuli were applied in all conditions, and we predicted the presence of visual effects of stimulus presentation (a factor common to all conditions in the visual system. In line with this expectation, we found robust event-related activity bilaterally in the fusiform cortex (see Fig. 1 for
Discussion
The primary aim of this study was to localize regions of the human brain that were responsive to violations of expectations related to freely delivered monetary rewards. Before scanning, subjects learned that a red circle would be followed by a rewarding outcome and a blue circle by a nonrewarding outcome. We introduced two types of prediction error by occasionally delivering outcomes that were unexpected: The failure of an expected reward (a nonrewarding outcome was delivered when rewarding
References (96)
- et al.
Functional imaging of neural responses to expectancy and experience of monetary gains and losses
Neuron
(2001) - et al.
Differential neural response to positive and negative feedback in planning and guessing tasks
Neuropsychologia
(1997) - et al.
Neural interaction between the basal forebrain and functionally distinct prefrontal cortices in the rhesus monkey
Neuroscience
(2001) - et al.
Monkeys can associate visual stimuli with reward delayed by 1 s even after perirhinal cortex ablation, uncinate fascicle section or amygdalectomy
Behav. Brain Res.
(1997) - et al.
FMRI visualization of brain activity during a monetary incentive delay task
NeuroImage
(2000) - et al.
A region of mesial prefrontal cortex tracks monetarily rewarding outcomes: characterization with rapid event-related fMRI
NeuroImage
(2003) - et al.
Cognitive and language functions of the human cerebellum
Trends Neurosci.
(1993) - et al.
Effect of expected reward magnitude on the response of neurons in the dorsolateral prefrontal cortex of the macaque
Neuron
(1999) - et al.
Distribution of dopaminergic receptors in the primate cerebral cortex: quantitative autoradiographic analysis using [3H]raclopride, [3H]spiperone and [3H]SCH23390
Neuroscience
(1991) - et al.
Reward mechanisms in the brain and their role in dependence: evidence from neurophysiological and neuroimaging studies
Brain Res., Brain Res. Rev.
(2001)