ReviewModel-Based Control in Dimensional Psychiatry
Section snippets
Measures of Goal-Directed and Habit Control
Measures of goal-directed and habit control in preclinical and human studies can be divided into two basic forms: conventional overtraining and devaluation tasks (Figure 1) 4, 5 and sequential decision tasks (also known as multistep tasks, with the most common being the two-step task) (Figure 2). Here we describe these concepts.
Goal-directed and habitual behaviors are common in daily decisions. We might see (stimulus) and take (response) the same turn-off when driving home (the goal). If,
Neural Substrates of Goal-Directed and Habit Tasks in Healthy Humans
Rodent and human studies implicate similar dissociable frontostriatal regions in the balance between goal-directed and habit learning. Lesions of the rodent dorsomedial striatum (human caudate) and prelimbic cortex block goal-directed behaviors, leaving intact habit learning 8, 9. In contrast, lesions of the dorsolateral striatum (human putamen) and infralimbic cortex result in intact goal-directed behaviors despite extended training 9, 10, 11.
Human functional magnetic resonance imaging studies
Relevance to Dimensional Psychiatry
The role of model-based control as a dimensional construct underlying compulsive behaviors has been examined across multiple psychiatric disorders. Compulsive behaviors can be defined as a rigid pattern of maladaptive behaviors despite negative consequences (24). We examine similar behavioral phenotypes, namely drug seeking despite negative consequences in addiction and repetitive binge eating phenomena in binge eating disorder (BED), both characterized by the pathological use of drugs or food.
Neurochemical Substrates of Multistep Tasks
In rodent studies, dopamine enhances habit formation (31) related to dopamine D1 receptor activation (53). Selective nigrostriatal dopaminergic lesions impair habit formation (54). However, in humans, a different picture emerges. Depleting the dopamine precursor with tyrosine depletion increases habitual control in the slips-of-action task (55). In Parkinson’s disease, characterized by greater dopaminergic depletion of dorsal relative to ventral striatal regions, impairments in habit control
Characteristics of Multistep Tasks
Several lines of evidence suggest that the model-based measure in the two-step task appropriately captures goal-directed control. Support for commonalities between the model-free measure and conventional overtraining and devaluation tasks exists but is more mixed.
The two-step task in healthy control subjects correlates with a conventional overtraining and food outcome devaluation task, which most closely approximates rodent models (66). However, factors commonly known to influence the shift
Pavlovian Learning: Model-Based Control and the Orbitofrontal Cortex
Beyond instrumental learning, the broader construct of inferred model-based control (reflecting imagined value based on knowledge of the associative task structure) versus model-free control (relying on stored values) is also relevant to pavlovian processes and is shown to be dissociable in rodent studies 45, 74, 75 (Figure 5). These processes may be particularly relevant for addictions in which drug-related cues may represent learned pavlovian associations between neutral and unconditioned
Conclusions
Model-based and model-free control in instrumental (goal-directed and habit learning) and pavlovian learning processes shows translational clinical relevance for dimensional psychiatry. The model-based measure in sequential decision tasks appears to formalize the measure of goal-directed control and is impaired across compulsive behaviors. Habit learning from overtrained and devaluation habit procedures have similarly shown enhancement in compulsive behaviors. The overlapping construct of
Acknowledgments and Disclosures
VV is funded by a Medical Research Council Senior Clinical Fellowship. MS and AR are supported by the German Research Foundation (Deutsche Forschungsgemeinschaft).
VV has received fees as an expert court witness for proceedings related to a dopamine agonist. AR, MS, and SG report no biomedical financial interests or potential conflicts of interest.
References (79)
- et al.
Model-based influences on humans’ choices and striatal prediction errors
Neuron
(2011) - et al.
States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning
Neuron
(2010) - et al.
Goals and habits in the brain
Neuron
(2013) - et al.
Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates
Neuropharmacology
(1998) - et al.
Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans
Neuron
(2013) - et al.
Fronto-striatal organization: Defining functional and microstructural substrates of behavioural flexibility
Cortex
(2016) - et al.
Repeated episodes of chronic intermittent ethanol promote insensitivity to devaluation of the reinforcing effect of ethanol
Alcohol
(2014) - et al.
Habitual alcohol seeking: Time course and the contribution of subregions of the dorsal striatum
Biol Psychiatry
(2012) - et al.
Post-training cocaine administration facilitates habit learning and requires the infralimbic cortex and dorsolateral striatum
Neurobiol Learn Mem
(2015) - et al.
Slave to habit? Obesity is associated with decreased behavioural sensitivity to reward devaluation
Appetite
(2015)
Enhanced avoidance habits in obsessive-compulsive disorder
Biol Psychiatry
Counterfactual processing of economic action–outcome alternatives in obsessive-compulsive disorder: Further evidence of impaired goal-directed behavior
Biol Psychiatry
Dopamine enhances model-based over model-free choice behavior
Neuron
Opponent interactions between serotonin and dopamine
Neural Netw
The interaction of acute and chronic stress impairs model-based behavioral control
Psychoneuroendocrinology
The impact of orbitofrontal dysfunction on cocaine addiction
Nat Neurosci
Instrumental responding following reinforcer devaluation
Q J Exp Psychol
The role of learning in the operation of motivational systems
Differential engagement of the ventromedial prefrontal cortex by goal-directed and habitual behavior toward food pictures in humans
J Neurosci
The role of the dorsomedial striatum in instrumental conditioning
Eur J Neurosci
Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning
Eur J Neurosci
Coordination of actions and habits in the medial prefrontal cortex of rats
Cereb Cortex
A specific role for posterior dorsolateral striatum in human habit learning
Eur J Neurosci
Changes in corticostriatal connectivity during reinforcement learning in humans
Hum Brain Mapp
Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control
J Neurosci
Mapping value based planning and extensively trained choice in the human brain
Nat Neurosci
Disorders of compulsivity: A common bias towards learning habits
Mol Psychiatry
Determining the neural substrates of goal-directed learning in the human brain
J Neurosci
Simple plans or sophisticated habits? State, transition and learning interactions in the two-step task
PLoS Comput Biol
Neural systems of reinforcement for drug addiction: From actions to habits to compulsion
Nat Neurosci
Dopamine, reinforcement learning, and addiction
Pharmacopsychiatry
Alcohol seeking by rats: Action or habit?
Q J Exp Psychol B
Shift from goal-directed to habitual cocaine seeking after prolonged experience in rats
J Neurosci
Amphetamine exposure enhances habit formation
J Neurosci
No association of goal-directed and habitual control with alcohol consumption in young adults
Addict Biol
Characterizing a psychiatric symptom dimension related to deficits in goal-directed control
eLife
Cited by (76)
Risk-promoting effects of reward-paired cues in human sign- and goal-trackers
2024, Behavioural Brain ResearchGoal-directed and habitual decision making under stress in gambling disorder: An fMRI study
2023, Addictive BehaviorsComputational Mechanisms of Addiction and Anxiety: A Developmental Perspective
2023, Biological PsychiatrySpatiotemporal Precision of Neuroimaging in Psychiatry
2023, Biological PsychiatryBelief updating in psychosis, depression and anxiety disorders: A systematic review across computational modelling approaches
2023, Neuroscience and Biobehavioral Reviews