Thread by @leighblue

Does the brain’s representation of imagined reward explain how the modern economy works? Some recent developments in psychology, neuroscience and economics suggest that it might. 1/25

We start from economics, and in particular a set of questions about non-material aspects of trade and economic value. Schelling, Loewenstein and others have asked why we value things (beliefs, the fates of characters on TV) that are wholly products of our own minds. 2/

To answer this, look at a psychological question that may at first seem unrelated. How are we able to defer current reward/pleasure in favour of future outcomes? How do we delay gratification? Economic theory posits a “discounting” relationship between present & future reward, 3/

...but doesn’t get into how that is actually implemented. If we believe that people (and/or their brains) are motivated by incentives or other causes, how can a benefit that won’t come for 30 years (my income in retirement) motivate my present self? Is my mind time travelling? 4/

Ainslie’s Picoeconomics hints at an option: does the brain reward me today for putting away money for my future? But again there’s no clear mechanism to explain why this would happen.

Enter a recent emerging consensus in neuroscience: the predictive brain. 5/

In this theory, the brain’s job is to continually predict the future and test present experiences against the predictions. It’s computationally impossible to accurately interpret all incoming sensory signals in real time and build a good picture of the environment around us. 6/

So instead the brain relies on a stable mental model of the world, using sensory signals to test, confirm or challenge its beliefs, and update the model when needed. There’s good evidence for this in phenomena like confirmation bias and optical illusions. 7/

The model is also being directly tested (eg by @neuro_kim and @jhamrick) in rodent brains and computational simulations. You can see more about this in @fluffycyborg’s work.

So the brain predicts the future. Then what? There’s a psychological concept called “prospection”. 8/

Prospection (see Gilbert & Wilson 2007) is the psychological experience of the future - mentally simulating what may happen to us. We habitually do this and (often) enjoy it. The process helps us make decisions (Pezzulo & Rigoli 2011) - simulate the outcome of option A and see 9/

how it feels, do the same for option B and pick the one that is most rewarding (Pham’s (1998) “How-Does-It-Feel” heuristic).

The predictive brain is usually discussed for near-term predictions - what will I see when I open this door? Prospection can be over multiple decades. 10/

But what if these are implemented by exactly the same mechanism? (Osman 2014 makes a similar argument regarding the multiple time horizons of our sense of agency and control.) I argue that prospecting and predicting generate reward, which can be “traded” for primary rewards. 11/

This “synthetic reward” generated by imagining future positive outcomes is what we use to defer immediate gratification. The imagined reward can outweigh the reward from consuming right now.

We also have some hints about the mechanism by which this process might work. 12/

Reinforcement learning is a well known mechanism by which reward can create motivation. Right back to Skinner’s operant conditioning and Pavlov’s classical conditioning. (RL is now often used in machine learning - I may come back to this in another thread). 13/

In 1997, Dayan, Schultz and Montague showed that brains can learn chains of stimulus-response pairs and dopamine response “migrates” up the chain. If I’m rewarded for eating chocolate, I’ll learn to be rewarded by seeing chocolate - or even just seeing the packaging or brand. 14/

The brain’s ability to simulate sensory experiences may arise here. I learn a causal chain, the brain rewards me for activating this causal chain and tracing its links. This is what enables me to choose a stimulus causally linked to high reward, over a low immediate reward. 15/

Stachenfeld’s hippocampal predictive map implementing Dayan’s successor representation is an example of this in recent neuroscience research. In psychology, Sloman & Lagnado’s causal graphs express a similar insight. In prospection, we navigate causal graphs and are rewarded. 16/

Why do we do this even when we aren’t evaluating a specific choice option? Because it’s intrinsically rewarding (a side-effect of the fact that we’re rewarded for evaluating deferred positive outcomes, a capability with immense evolutionary value knowingandmaking.com/2017/05/the-am… ). 17/

One consequence is that the brain can learn shortcuts to reward. By evaluating these causal chains (A-B-C-D, where D is the primary reward from the chocolate) I can learn that A (the sweet shop) is rewarding without consciously following the chain. I “cache” the reward at A. 18/

This provides a powerful computational shortcut, the difference between model-based representations (which are very flexible) and model-free (which are very efficient). The brain continually daydreams, evaluating these causal chains and learning cached rewards for future use. 19/

A happy side-effect: as we learn bigger causal networks we apply them to the actions of other people (it’s valuable for us to be able to predict these) and gain synthetic reward by imagining good outcomes for them. Strategically helpful and also implements empathy! 20/

Those other people don’t need to be real. We get reward from imagining fictional outcomes too. Enjoy watching a movie? It’s the same predictive reward mechanism at work. More generally, it’s reward from representation, not just prediction. And we cache those representations. 21/

So cached representations which originated as the start of causal chains, become intrinsically rewarding. This is why we value beliefs, or symbols, or imagined states: because they once predicted rewarding outcomes. The brain has to take shortcuts, so it forgets the chain. 22/

Today’s economy (and politics), at least in rich countries, is dominated by such symbols and beliefs. Material survival, for most, is not the daily concern (though for some, the symbols and survival are linked). So the task is to develop economic models that reflect this. 23/

This is much more than behavioural economics: it’s an economics of symbols, of reward, of intermediate goods, and it impacts the theory of labour, consumption, growth, and maybe money. I call it cognitive economics and the psychological mechanisms underlying it, “System 3”. 24/

For citations to most of the work mentioned here: knowingandmaking.com/2018/04/neuros… And you can come and see me present results of some experiments on this at TIBER next month: tilburguniversity.edu/upload/d3175ad…