Thread by @mcmullarkey

About to live tweet "Recent Advances in the Use of Modeling to Explain and Predict Psychological Phenomena From Nomothetic & Idiographic Perspectives" with @EikoFried @talyarkoni @DepressionLab @aaronjfisher #aps19dc

It's already won the award for longest title, so good start!

@EikoFried @talyarkoni @DepressionLab @aaronjfisher Twitter-less (I think!) Don Robinaugh and Jonas Dalege are also presenting

@EikoFried @talyarkoni @DepressionLab @aaronjfisher .@EikoFried starts us off by reminding us that psychological modeling are complex, multicausal constructs and our approaches to these constructs often don't match that complexity

@EikoFried @talyarkoni @DepressionLab @aaronjfisher Then, mentions that many of our perspectives are about explaining constructs, but perhaps we should focus more on predicting key aspects of them

And also, Nomothetic (group level) vs. Idiographic (individual level) analyses! We'll look at both

Starting off with Don Robinaugh talking about a computational model of panic disorder

Our theories of panic disorder haven't really advanced in 30 years

Formalizing theories using mathematical models might help advance theory

Can more specifically allow us to examine what the theory can and can't explain (Find hidden assumptions)

So, they tried to model the vicious cycle theory of panic disorder

Need to go beyond saying there's a causal effect between these variables, we need to say what kind of relationship there is

Defines the rate in change of arousal as a function of itself and perceived threat (originally modeled as linear)

Key: The more you believe arousal is dangerous the more you'll perceive arousal as dangerous

Can add this moderation to the equation and trace paths through something called a vector field

This means that if we have an equation we can figure out where the system will end up as long as we know the inputs

The model then got a lot more complex by adding escape behaviors, avoidance, and learning

They mathematically modeled the classic CBT idea that if you engage in escape behaviors you don't learn arousal isn't dangerous

But can this theory explain panic attacks?

If you don't believe arousal is dangerous, you see variations in arousal not leading to changes in perceived threat

But if you do panic attacks (high perceived threat) occur spontaneously and non-predictably

They can also predict panic disorder in a similar way over a 12 week simulation (previous simulation was over 12 hours)

But it didn't account for non-clinical panic attacks, as everyone simulated under this model who experienced a panic attacks developed panic disorder

But taking this computational approach allows us to far more easily recognize this limitation! And it gives us a an explicit direction for future research

Now Jonas Delange is talking about applying a more advanced theory (entropy) to a younger science (attitude networks)

We have to think about both micro and macro states

Micro = Exact configuration of attitude elements
Macro = Overall attitude

Systems high in entropy can be realized by many different microstates

For attitudes, we can not like someone if we think they're smart but not nice OR nice but not smart (Potentially high entropy system)

An attitude that's highly fluctuating is like water as a gas, high entropy

An attitude that's relatively fixed is like water as a solid, low entropy

And the 2nd law of thermodynixs = Entropy always increases

We'd maybe like lower entropy attitudes (Note from me: though can think of times where we'd want them to be more dynamic, e.g., maybe I don't hate myself...)

Can think of attitudes as networks with nodes and edges. Specifically, an Ising network where microstates of attitudes are either "on" or "off"

There's a bunch of effects that follow from their theory. Check out the paper for all of them!

Paper is "The Attitudinal Entropy Framework as a general theory of individual attitudes" in Psychological Inquiry with Dalege as lead author

Aaron Fisher (@aaronjfisher)promises no equations! He also has made all of his data on OSF (it's his pinned tweet right now)

@aaronjfisher Precision medicine was primarily based on genes, but it hasn't panned out in psychology so far

His pitch: Personalized medicine can use behavioral data to make behavioral predictions to alleviate behavioral problems

His bigger pitch: Human behavior is predictable

He says making one model at the group level won't work, but we should build a pipeline where the process is generalizable but the set of predictors will be unique for each individual

Also, the dependent variables matter too! He'll talk more later

To define predictors, did lit review, then did focus groups to make sure people vibed with them

Did this to try to predict smoking behavior, but could do this for almost any type of behavior

We're talking about trends, cycles, and recurrences

Trends are shifts in mean level and non-repeating, and they can mess up this type of work

Cycles and recurrences repeat at regular intervals, and we can put these in the model

(Didn't put trend in time series, excluded people with non-stationary data, or data with a trend)

They used elastic net regularization which knocks out predictors (or can shrink them toward each other, though I'm not sure if his version did that)

Also used Naive Bayes Classification, which doesn't knock out predictors

These are lagged predictions, so they're predicting whether someone will or won't smoke in the next few hours

They split it 75/25 into training and testing

Used Area Under the Curve as accuracy, elastic net gets 0.70-0.75, naive Bayes gets 0.78

He says solid C, C+

2 people they predicted just at chance, but 2 people they predicted whether or not they would smoke with 100% accuracy

Question: Why not benchmark against individual smoking frequency or the smoking on the first day vs. a 50% chance of smoking as implied by AUC models?

Says that wouldn't work, recommends checking out paper

.@DepressionLab is talking heterogeneity in statistical and conceptual approaches to predicting treatment outcomes

They did a tournament (The SMART tournament) where teams competed to best predict who would respond to treatment

They had a 4,000 person training set, 2,000 person test set

Asked everyone to predict how well people would respond to both low and high intensity treatment

There weren't very many variables collected for each person (features in machine learning speak), but they are used to screen over a million people a year in the UK

So, were people able to predict response to treatment?

High bar, as the data was feature poor, heteogenous presentations, and heterogenous treatment types

The teams made wildly different predictions about who would and wouldn't respond

Most teams outperformed the baseline model in predicting low intensity treatment response, but most teams didn't predict high intensity treatment response that well

But the teams that did well did really well!

Also, if we can identify people who are likely to respond to any kind of intensity of treatment, let's give them lower intensity! And if they're likely to not respond to low intensity, let's step them up ASAP

Most models didn't generalize well, but a team that used Bayesian Additive Regression Trees (I think!) had models that generalized well to 5 other sites that weren't included in the original training or test data

Now @talyarkoni is doing discussant stuff

Focusing on one issue: Embracing complexity

One perspective: We need to think more carefully about the kinds of models that can fit in our heads

Another perspective: We might create some kind of incredibly complex model that no one can understand but makes good predictions

It's easier to reason about explanatory models, but the road from the theory in the head to an outcome is long

In machine learning, the road from the model to the outcome can be much shorter

But these complex predictive models can be very brittle and generally don't generalize

The neutral response is that there's no right answer. This is a case by case issue. Panic disorder model looks great, but would that work for schizophrenia?

Less neutral: In psychology we are way tilted toward explanatory vs. predictive perspectives

He's talking about GPT2

A language model introduced two months ago to predict text

It can write stories from a prompt

It wrote a dope story about unicorns that sounded amazing

What's the relevance to clinical psych?

5 years ago: Is it easier to create this story than predicting treatment response? Everyone would say treatment response

So if you're a funder: Do you really want 20-30 years of maybe that's well reasoned vs. a model that can't fit in our heads but predicts outcomes well

We don't know how we make lots of everyday decisions, we're just looking for safeguards against failures