Invited ReviewNeural substrates underlying effort, time, and risk-based decision making in motivated behavior
Introduction
Some of the earliest laboratory studies of motivated behavior led researchers to observe that most complex behavior tends to occur in bouts and that specific behaviors such as feeding or grooming can be characterized by their frequency, intensity, temporal distribution and direction towards or away from a particular stimulus. One of the prominent researchers of the day went so far as to say that identifying the factors responsible for the initiation and termination of these specific bouts of behavior would be the central problem for experimental psychologists to understand (Richter, 1927). Over the years there have been numerous theories of motivation put forth (Bolles and Moot, 1972, Hebb, 1955, Hull, 1943, Young, 1961), each of which has been influential in stimulating what has been a continuous stream of experiments and research on this topic. There exist excellent reviews of many of these theories and concepts (Berridge, 2004).
Almost a century later, researchers from numerous fields including psychology, psychiatry, and neurobiology are still actively studying goal-directed motivation, which is the name that has been given to the set of biological and psychological processes which guides behavior in pursuit of a goal. Research in this realm of behavioral neuroscience has come a long way toward understanding the wide array of factors which come together to modulate goal-directed action. Neurobiologists are uncovering the widely distributed collection of neural circuits which underlie the various aspects of goal-directed motivation. This has led to the identification of limbic and midbrain regions including the Ventral Tegmental Area (VTA), Nucleus Accumbens (NAcc), and Ventral Pallidum (VP) which appear to be critical for invigorating effortful behavior. Additionally, cortical regions such as the Anterior Cingulate Cortex (ACC) and medial Prefrontal Cortex (mPFC) are crucial for comparing costs and benefits which becomes important when one is faced with several potential response choices. In addition to the basic work being done in animal models, clinicians and psychiatrists using modern brain imaging methods have started to uncover some of the neurobiological correlates of impairments in goal-directed motivation commonly seen in many forms of psychopathology, including schizophrenia and depression. Currently, the unprecedented technical arsenal of neuroscience tools available to researchers makes it an extremely exciting and fruitful time to be studying a question which has captivated researchers for nearly a century.
All mobile organisms are faced with the universal challenge of living in a world in which the resources needed for survival may be limited in number and unevenly dispersed throughout the environment. Obtaining essential resources often requires one to overcome obstacles which inherently contain many different kinds of costs to the organism. When seeking food, water, or potential mates, one might be faced with any number of these costs, including: a physical distance one must traverse, the height of an obstacle one must climb, the number of responses one must make, or the commitment of time one must invest. Goal-directed motivation represents the set of processes which allows an organism to weigh these costs against potential benefits of obtaining a goal. It has been recognized by researcher for a long time that motivation serves two important functions, as it provides both a directional influence on behavior and also has an activational or energizing effect as well as (Duffy, 1957, Hebb, 1955); and more recent work has started to describe the underlying neurobiological substrates of both the directional processes (Kim et al., 2013, Kimchi and Laubach, 2009) as well as activational processes (Anaclet et al., 2009, Pfaff et al., 2012) and whereas the directional component of motivation guides behavior toward a specific goal and away from competing actions (Dickinson & Balleine, 1994), the activational component of motivation provides the energy or vigor needed to overcome the physical costs standing between the animal and its goal. This activational influence on motivation is reflected in the likelihood of initiation, and the speed, vigor and persistence of an action (Floresco, 2015, Salamone, 1992, Salamone and Correa, 2002, Salamone et al., 2012).
The most general way in which the concept of directional motivation is used is to say that animals pursue positive stimuli (e.g. food, water, sex, etc.) and avoid negative stimuli (e.g. painful conditions, predators, stress) (Salamone, Yohn, López-Cruz, San Miguel, & Correa, 2016). A more specific definition of the concept of directional motivation is the processes which cause animals to choose one specific class of behavior to engage in at a given time over all others (i.e. Feeding, Drinking, Mating, Aggressive Behavior, etc.). This concept proves useful in that it allows researchers to attempt to figure out the physiological and environmental variables which influence animals to engage in one class of behaviors over another (e.g. feeding as opposed to drinking). This usage helps to explain observations such as when animals choose to pursue food following a long period of food deprivation, as it is the directional influence of motivation which leads the animal to pursue food while forgoing pursuits of other behaviors. This is unsurprising as there are distinct neural circuits which control food seeking as opposed to something like thirst (Kelley et al., 2005, Oka et al., 2015). There has been an extensive amount of research aimed at understanding what circulating hormones and brain regions are responsible for directional motivational effects for feeding (Belgardt, Okamura, & Brüning, 2009), thirst (Johnson & Thunhorst, 1997), as well as sexual behavior (Davidson, 1966), and other social behaviors (Hong et al., 2014, Wang et al., 2014). We point readers to recent reviews of this literature (Sternson, 2013), as an extensive discussion of these directional effects are beyond the scope of the present review. In the present review, we focus on situations in which subjects are food restricted and working for food rewards (i.e. experimentally manipulated to be directed towards food), and we examine how different types of costs a subject must overcome to obtain the food reward alters both activational aspects of behavior and the choice of what specific action to take to obtain reward.
As animals are deprived of necessary resources their behavior changes in a number of ways: (a) there is often an increase in general locomotor activity, (b) an increase the likelihood of performing actions known to lead to that deprived resource, (c) and an increase in the speed, vigor, and the persistence of these goal directed actions (Floresco, 2015, Salamone, 1992, Salamone and Correa, 2002, Salamone et al., 2012). These changes in behavior are thought to reflect changes in the activational or energizing effects of motivation. It is this activational or energizing influence of motivation which allows animals to overcome the costs standing between them and the goal for which they are working. In this review, we focus specifically on what is known about the neural substrates that influence how the costs of responding affect the activational aspects of motivated behavior. We also examine what is known about the neural machinery involved in processing information about different types of costs that enter into the cost-benefit computation that guides choices about how to allocate effort in situations in which there is more than one response option that could lead to the desired resource.
How does motivation properly guide an organism through the environment to overcome obstacles and meet needs necessary for survival? Current theories suggest that animals incorporate information from many different levels and perform cost-benefit computations which allow for adaptive decision making. A typical laboratory experiment in which a rat has learned to press a lever for a food reward serves as an excellent example of how this might work. A fully sated rat will make a very small number of lever presses for food. The few lever presses it does make will be made slowly with many pauses in between presses, and the rat will spend a substantial amount of time engaging in other behaviors such as exploring the chamber and grooming itself. The same animal’s behavior will look very different when its access to food has been restricted. Both the number of lever presses made as well as the rate/vigor of those responses are highly correlated with the percent body weight loss induced by the food restriction (Collier, 1969, Collier and Levitsky, 1967, Marwine and Collier, 1971). In these two scenarios the cost of responding is constant (i.e. the same number of lever presses is required in both situations), but the benefit or value of the food differs greatly. The difference between the cost and the benefit of pressing in each particular condition determines the direction of behavior (lever pressing and not exploring or grooming/etc.) as well as the intensity or vigor (response rate of the lever presses) with which the behaviors are executed.
Research over that last 5 decades shows that there are many factors which influence the cost-benefit decision making processes. These factors include environment factors (such as local food availability, time of day, or temperature), an animal’s experiential history (whether it was trained on a continuous or intermittent schedule of reinforcement), physiology (circulating hormone levels) and internal biological clocks (e.g. location in a circadian rhythm (Antle & Silver, 2015). Fig. 1 illustrates a conceptual model of how all of these factors might act in concert in a hierarchical manner to modulate goal-directed motivation by influencing the underlying cost-benefit decision making processes, and provides examples of these different factors influencing motivation (Simpson & Balsam, 2016). As shown in this figure, this model posits that the physiological state of the organism, the environment, and past history/learning of the organism interact to influence the representation of costs and benefits that determine the specific types of behavior at any given time. Moreover, the information about the costs and benefits are compared in a cost-benefit computation which then influences the selection and vigor of behavior. We present Fig. 1 to suggest one possible model of how goal-directed motivation may work, and to provide a context in which to place this review. We do not attempt to state which brain regions are definitively involved in specific stages of the Cost/Benefit computation process, rather we examine an array of studies which focus on the cost input to this computation. In doing so we compare 3 different kinds of costs: Effort, Time, and Risk. We bring together these three separate lines of investigation to identify both the overlapping and distinct neurobiological substrates for processing these costs.
The purpose of this review is to summarize and synthesize a number of varying studies which examine different types of motivated behavior through the framework of motivated behavior as relying on a cost-benefit computation to give rise to both the direction and vigor of behavior. The direction and vigor of behavior represent the final behavioral output which one can measure, and a number of studies are reviewed which have been performed to understand the neural locations at which manipulations to the region impact either directional or activational aspects of behavior. Additionally, we give a primary focus to studies which have examined motivated behavior through various forms of cost-benefit decision making. In this review, we systematically focus on studies which have manipulated one of the factors which goes into the cost-benefit calculation: cost, as this represents one critical side of the cost-benefit computations that guide motivated behavior. We first provide a summary of the behavioral data that demonstrates animals’ ability to process information related to various types of costs. We then discuss the more recent work examining the neurobiology of the activational effects of motivation. We finish by reviewing an array of studies aimed at understanding the neurobiological underpinnings of cost-benefit decision making by specifically focusing on studies which employed manipulations of three types of response costs: (1) effort, (2) time, and (3) risk. In doing so we describe studies which have employed neural manipulations such as various types of lesions, as well as locally delivered pharmacological manipulations. While we also discuss a number of results from systemic pharmacology studies, we have limited this to results which further inform our understanding of the neural circuits underlying the different behavioral processes discussed in the review.
Section snippets
Evidence of animals processing and using information about the costs going into cost-benefit computations underlying motivation
Motivation activates and directs behavior allowing organisms to overcome response costs to obtain specific goals. The decision to continue exerting effort in pursuit of a goal while neglecting other available response options is thought to be influenced by an underlying cost-benefit decision making process. During this process, the organism is thought to use information and knowledge of the costs of the current situation and weigh them against the anticipated benefit the effort will ultimately
Activational component of motivation can be observed through measures of response vigor/persistence
There are a number of different tasks which have allowed researcher to quantify changes in response vigor/persistence. Many of the tasks which have been used involve having animals make responses of a single type to obtain the goal (i.e. running down a runway, or responding on a single lever). The activational component of motivation is readily observed in runway tasks as animals run faster for a food reward as a function of the duration that they have been deprived of food, or as a function of
Neurobiology of cost-benefit decision making: Manipulations of different costs
Over the last several decades a substantial amount of progress has been made toward understanding of the neural circuits involved in various aspects of cost-benefit decision making (Table 2). Many of the studies involved tasks which require subjects to make a choice between different response options. In these paradigms, animals have been faced with alternatives associated with different costs - differences in the effort requirements, the time delay from response choice to reward delivery, and
Summary and future directions
We have summarized a wide range of studies which address the question: how does the brain process and use information related to different types of response costs underlying motivated behavior? We focused on cost manipulations of effort, time delays, and risk/probability and found that a number of brain regions seem to appear to be important in many different types of tasks, whereas others appear to be highly specific to some tasks but not others.
Acknowledgements
This work was supported by the National Institute of Mental Health (NIMH) Grant 1R21MH104718 (E.H.S) and 5R01MH068073 (P.D.B).
References (206)
- et al.
Nucleus accumbens dopamine depletions make rats more sensitive to high ratio requirements but do not impair primary food reinforcement
Neuroscience
(1999) - et al.
Effects of dopamine antagonists and accumbens dopamine depletions on time-constrained progressive-ratio performance
Pharmacology Biochemistry and Behavior
(1998) - et al.
Meal patterns of mice under systematically varying approach and unit costs for food in a closed economy
Physiology & Behavior
(2009) - et al.
Effects of selective dopamine D1 or D2 receptor blockade within nucleus accumbens subregions on ingestive behavior and associated motor activity
Behavioural Brain Research
(2002) - et al.
Temporal maps and informativeness in associative learning
Trends in Neurosciences
(2009) - et al.
Rats with different profiles of impulsive choice behavior exhibit differences in responses to caffeine and d-amphetamine and in medial prefrontal cortex 5-HT utilization
Behavioural Brain Research
(2008) - et al.
D1-like and D2 dopamine receptor antagonists administered into the shell subregion of the rat nucleus accumbens decrease cocaine, but not food, reinforcement
Neuroscience
(2005) Motivation concepts in behavioral neuroscience
Physiology & Behavior
(2004)- et al.
Effect of disconnecting the orbital prefrontal cortex from the nucleus accumbens core on inter-temporal choice behaviour: A quantitative analysis
Behavioural Brain Research
(2008) - et al.
Effect of quinolinic acid-induced lesions of the subthalamic nucleus on performance on a progressive-ratio schedule of reinforcement: A quantitative analysis
Behavioural Brain Research
(2008)
Schedule-dependent effects of haloperidol and amphetamine: Multiple-schedule task shows within-subject effects
Pharmacology Biochemistry and Behavior
Dopamine in the orbitofrontal cortex regulates operant responding under a progressive ratio of reinforcement in rats
Neuroscience Letters
Motivational responses to natural and drug rewards in rats with neonatal ventral hippocampal lesions: An animal model of dual diagnosis schizophrenia
Neuropsychopharmacology
Who is in charge? Animal vs experimenter control
Appetite
Substantia nigra pars reticulata GABA is involved in the regulation of operant lever pressing: Pharmacological and microdialysis studies
Neuroscience
Nucleus accumbens dopamine depletions alter relative response allocation in a T-maze cost/benefit task
Behavioural Brain Research
Nucleus-accumbens dopamine depletions in rats affect relative response allocation in a novel cost/benefit procedure
Pharmacology Biochemistry and Behavior
Different effects of nucleus accumbens and ventrolateral striatal dopamine depletions on instrumental response selection in the rat
Pharmacology Biochemistry and Behavior
Effects of reinforcer quality and step size on rats’ performance under progressive ratio schedules
Behavioural Processes
Behavioural changes following lesions of the orbital prefrontal cortex in male rats
Behavioural Brain Research
Dopamine, neuroleptics and reinforced behavior
Neuroscience & Biobehavioral Reviews
Forebrain circuitry involved in effort-related choice: Injections of the GABA(A) agonist muscimol into ventral pallidum alter response allocation in food-seeking behavior
Neuroscience
Nucleus accumbens and effort-related functions: Behavioral and neural markers of the interactions between adenosine A(2A) and dopamine D-2 receptors
Neuroscience
Lack of effect of prefeeding on food-reinforced temporal response differentiation and progressive ratio responding
Behavioural Processes
Preverbal and verbal counting and computation
Cognition
Nucleus accumbens dopamine depletions and time-constrained progressive ratio performance: Effects of different ratio requirements
Pharmacology Biochemistry and Behavior
Antagonistic control of social versus repetitive self-grooming behaviors by separable amygdala neuronal subsets
Cell
The neuroendocrinology of thirst and salt appetite: Visceral sensory signals and mechanisms of central integration
Frontiers in Neuroendocrinology
Transient and selective overexpression of dopamine D2 receptors in the striatum causes persistent abnormalities in prefrontal cortex functioning
Neuron
Corticostriatal-hypothalamic circuitry and food motivation: Integration of energy, action and reward
Physiology & Behavior
The effect of orbital prefrontal cortex lesions on performance on a progressive ratio schedule: Implications for models of inter-temporal choice
Behavioural Brain Research
Investigating striatal function through cell-type-specific manipulations
Neuroscience
Anterior cingulate cortical lesion attenuates food foraging in rats
Brain research bulletin
Organization of catecholamine neurons projecting to the frontal cortex in the rat
Brain Research
Orexin/hypocretin and histamine: Distinct roles in the control of wakefulness demonstrated using knock-out mouse models
The Journal of Neuroscience
Circadian insights into motivated behavior
A detailed behavioral analysis of the acute motor effects of caffeine in the rat: Involvement of adenosine A1 and A2A receptors
Psychopharmacology (Berlin, Germany)
A novel strategy for dissecting goal-directed action and arousal components of motivated behavior with a progressive hold-down task
Behavioral Neuroscience
Time and associative learning
Comparative Cognition & Behavior Reviews
Dopamine modulates effort-based decision making in rats
Behavioral Neuroscience
Enhanced food-related motivation after bilateral lesions of the subthalamic nucleus
Journal of Neuroscience
The subthalamic nucleus exerts opposite control on cocaine and ‘natural’ rewards
Nature Neuroscience
Timing behavior as a function of amount of reinforcement
Psychonomic Science
Hormone and glucose signalling in POMC and AgRP neurons
The Journal of Physiology
Effect of quinolinic acid-induced lesions of the nucleus accumbens core on performance on a progressive ratio schedule of reinforcement: Implications for inter-temporal choice
Psychopharmacology (Berl)
Quantitative analysis of the effect of lesions of the subthalamic nucleus on intertemporal choice: Further evidence for enhancement of the incentive value of food reinforcers
Behavioural Pharmacology
Instrumental learning in animals: Parameters of reinforcement
Annual Review of Psychology
Derived motives
Annual Review of Psychology
Reward magnitude and learning in a single-presentation discrimination
Journal of Comparative and Physiological Psychology
Impulsive choice induced in rats by lesions of the nucleus accumbens core
Science
Cited by (86)
Effort-based decision making and motivational deficits in stroke patients
2024, Brain and CognitionAccumbal adenosine A<inf>2A</inf> receptor inactivation biases for large and costly rewards in the effort- but not delay-based decision making
2023, NeuropharmacologyCitation Excerpt :We found that NAc-A2AR KD preferentially modulates the choice in E-CBDM, but not in D-CBDM. This differential regulation by NAc-A2ARs maybe attributed to the fact that E-CBDM and D-CBDM have distinct behavioral features/presentation and dissociable neurochemical mechanisms and neural circuits (Bailey et al., 2016; Floresco et al., 2008). Previous studies of primate species have shown that a willingness to expend more effort does not correlate with an inclination to tolerate delay costs to obtain a greater reward (Stevens et al., 2005a, 2005b), implying that delay- and effort-based CBDM are controlled by independent neural systems in the brain.