Is applied behavioural science stuck at a local maximum?
My sense is that there is a set of technical and ethical limitations that are getting in the way of progress.
Here is how I'm currently thinking about these limits and ways to overcome them (when possible).
My sense is that there is a set of technical and ethical limitations that are getting in the way of progress.
Here is how I'm currently thinking about these limits and ways to overcome them (when possible).
Firstly, the Technical Limitations:
1) Replicability
2) Unknown Boundary Conditions
3) Complementary & Crowding-Out Effects
4) Cultural Variation
5) With-in and Between-subject Idiosyncrasies
6) Channel & Data Access
7) Implementation Issues
8) Unforeseeable Second-Order Effects
1) Replicability
2) Unknown Boundary Conditions
3) Complementary & Crowding-Out Effects
4) Cultural Variation
5) With-in and Between-subject Idiosyncrasies
6) Channel & Data Access
7) Implementation Issues
8) Unforeseeable Second-Order Effects
(1/8) Replicability
A barrage of replication failures has called behavioural research findings, and the methodologies being used, into question.
Pre-registration, larger samples, field experimentation and research replication incentives are being used to overcome this 💪
A barrage of replication failures has called behavioural research findings, and the methodologies being used, into question.
Pre-registration, larger samples, field experimentation and research replication incentives are being used to overcome this 💪
(2/8) Unknown Boundary Conditions
Okay, this one's a bit more tricky. Interventions are more effective given the presence of certain conditions. These conditions aren't well understood yet.
Progress is made here by studying a variable across a range of contexts + meta-analyses.
Okay, this one's a bit more tricky. Interventions are more effective given the presence of certain conditions. These conditions aren't well understood yet.
Progress is made here by studying a variable across a range of contexts + meta-analyses.
(3/8) Complementary & Crowding-Out Effects
This is similar to boundary conditions I suppose. Practitioners often use a combo of interventions. Is more better than less? Maybe. Maybe not.
Studying the combinatory effects of interventions (e.g WOOP) is the way forward here.
This is similar to boundary conditions I suppose. Practitioners often use a combo of interventions. Is more better than less? Maybe. Maybe not.
Studying the combinatory effects of interventions (e.g WOOP) is the way forward here.
(4/8) Cultural Variation
This one is close to home for me. I work in Africa, yet lean on US/UK research. The challenge? Cultural factors (e.g. norms) can mediate the effects of interventions in unexpected ways.
To solve: Treat findings as hypotheses and run experiments locally.
This one is close to home for me. I work in Africa, yet lean on US/UK research. The challenge? Cultural factors (e.g. norms) can mediate the effects of interventions in unexpected ways.
To solve: Treat findings as hypotheses and run experiments locally.
(5/8) With-in and Between-subject Idiosyncrasies
Individuals within a seemingly similar context are more or less responsive to interventions for unknown reasons, and their responsiveness changes over time.
Data access + ML and behavioural profiling are making some inroads here.
Individuals within a seemingly similar context are more or less responsive to interventions for unknown reasons, and their responsiveness changes over time.
Data access + ML and behavioural profiling are making some inroads here.
(6/8) Channel & Data Access Limitations
The effectiveness of interventions can be constrained by the channels available to the practitioner. In addition, data on the target behaviour can be hard to collect, leading to a reliance on proxy measures.
These are tricky to overcome🤔
The effectiveness of interventions can be constrained by the channels available to the practitioner. In addition, data on the target behaviour can be hard to collect, leading to a reliance on proxy measures.
These are tricky to overcome🤔
(7/8) Implementation Issues
Even if a robust behavioural research finding is trialled and proven to be effective within a local context, it can still fail when implemented at scale.
The scaling scenario should be mapped out before the test, examined and resources identified.
Even if a robust behavioural research finding is trialled and proven to be effective within a local context, it can still fail when implemented at scale.
The scaling scenario should be mapped out before the test, examined and resources identified.
(8/8) Unforeseeable Second-Order Effects
Interventions don't affect behaviour in a vacuum. There can be serious side effects and externalities. These are difficult to map, let alone measure.
Premortems and systems thinking tools are useful for mapping here. Measuring is tricky.
Interventions don't affect behaviour in a vacuum. There can be serious side effects and externalities. These are difficult to map, let alone measure.
Premortems and systems thinking tools are useful for mapping here. Measuring is tricky.
So robust research methods, larger samples, fieldwork across different contexts, meta-analyses, localised testing, channel expansion, individual-level tracking, ML and externality measures get us started.
However, these need to be considered alongside the ethical limitations.
However, these need to be considered alongside the ethical limitations.
The Ethical Limitations:
1) Preference Ambiguity
2) Intention Misattribution
3) Lack of Transparency
4) Pseudo-Reversibility
5) Unknown Externalities
6) Data Tracking and Privacy Issues
7) Broader Psychological Implications
1) Preference Ambiguity
2) Intention Misattribution
3) Lack of Transparency
4) Pseudo-Reversibility
5) Unknown Externalities
6) Data Tracking and Privacy Issues
7) Broader Psychological Implications
(1/7) Preference Ambiguity
In some situations, it may be relatively easy to infer what an individual's preferences are. However, there are many scenarios where it's not so clear.
Asking people helps. But that's often not possible or practical given personalisation constraints.
In some situations, it may be relatively easy to infer what an individual's preferences are. However, there are many scenarios where it's not so clear.
Asking people helps. But that's often not possible or practical given personalisation constraints.
(2/7) Intention Misattribution
Sometimes the overarching value of an individual is clear (good health), and this leads to the assumption that they intend to perform an action (getting a vaccine).
This kind of misattribution of intention means we can't rely on revealed values.
Sometimes the overarching value of an individual is clear (good health), and this leads to the assumption that they intend to perform an action (getting a vaccine).
This kind of misattribution of intention means we can't rely on revealed values.
(3/7) Lack of Transparency
Some behaviourally-informed interventions (by their nature) work better when the targeted individual isn't aware of them.
Another issue here is that the intervention may be known, but the mechanism doing the work is opaque to the individual.
Some behaviourally-informed interventions (by their nature) work better when the targeted individual isn't aware of them.
Another issue here is that the intervention may be known, but the mechanism doing the work is opaque to the individual.
(4/7) Pseudo-Reversibility
Because behavioural interventions often operate outside of conscious awareness, even when optionality is available, the ability to reverse a choice may not always be possible, practically speaking.
Because behavioural interventions often operate outside of conscious awareness, even when optionality is available, the ability to reverse a choice may not always be possible, practically speaking.
(5/7) Unknown Externalities
Even if it is clear that an intervention has a positive effect on a behaviour (that an individual knowingly intends to perform), there may be second-order effects that outweigh the value created (e.g. pendulum or moral licensing effects).
Even if it is clear that an intervention has a positive effect on a behaviour (that an individual knowingly intends to perform), there may be second-order effects that outweigh the value created (e.g. pendulum or moral licensing effects).
(6/7) Data Tracking and Privacy Issues
Behavioural, linguistic and psychographic data can improve the impact of an intervention. However, this data can be (and has been) collected without the individuals' knowledge or consent, to influence mental representations and behaviour.
Behavioural, linguistic and psychographic data can improve the impact of an intervention. However, this data can be (and has been) collected without the individuals' knowledge or consent, to influence mental representations and behaviour.
(7/7) Broader Psychological Implications
Feedback I often see is that the worldview presented by behavioural science leans more towards the paternal than the liberal. This psychological implications of this belief-set shouldn't be overlooked, given the research on the subject.
Feedback I often see is that the worldview presented by behavioural science leans more towards the paternal than the liberal. This psychological implications of this belief-set shouldn't be overlooked, given the research on the subject.
It is less obvious to me what the solutions to these ethical limitations are.
Additionally, some of them seem destined to be exacerbated in an attempt to overcome the technical limitations (e.g. individual-level data collection).
Additionally, some of them seem destined to be exacerbated in an attempt to overcome the technical limitations (e.g. individual-level data collection).
Understanding these limits is the reason I have decided to dedicate attention to exploring alternative approaches and intersections that may bear fruit with time.
The path that I'm the most optimistic about at the moment self-initiated behavioural science. More on this soon.
The path that I'm the most optimistic about at the moment self-initiated behavioural science. More on this soon.
Am I missing any limitations or ways to resolve them? I'd be very interested in getting others thoughts on this.
