Thread by @suzyjstyles

Ah I really enjoyed being Discussant for today’s symposium at #SPR2022 @TheRealSPR
It’s a joy to bring these three projects together and talk about the challenges of research using high dimensional neuro data and some pathways to doing better!
1/

@VanjaKovic led us off with an intro to the three great projects being brought together for this syposium, highlighting their focus on building open communities to advance research

@angelasoskic shared the background behind the ARTEM-IS project — reminding us that even within one rather narrow study type (N400s) there can be as many analysis pipelines as there are papers!

Link: doi.org/10.1007s11065-…

but at the same time, not everybody writes down all of the steps involved in their research - so how can we know which pathway or pipeline to follow? This both makes replication and meta-science hard.

We have seen attempts at publishing ‘guidelines’ for reporting, but they don’t make much difference. the ARTEM-IS idea is that we might be able to build tools that help people do the job of reporting more easily! And we can do this job better together with others!

@angelasoskic also shared a walk through of some of the features of our ARTEM-IS Web App which is ready for you to play around with! In the next stage of Dev one of the goals is to seek user feedback about how we can make it even easier to use!

Try it out
psihologija.ff.uns.ac.rs/erp/

The the app was developed with a series of Design Guidelines in mind - to help make sure that the tool would be intuitive to use and fool-proof by design!
More about the design process here:
doi.org/10.1016/j.neur…

… and the INCF Working Group is open for new members to join the team!

incf.org/sig/incf-worki…

Next, @ugpavlov presented on the #EEGManyLabs project - another project tackling uncertainty in the published EEG literature - This time by bringing a community together for a series of replication studies -

sciencedirect.com/science/articl…

In the next talk @nastalfischer introduced the background behind #EEGManyPipelines which came out of a Twitter poll 🙀
Will EEG studies analysed in 70 different ways be more or less variable than the chaos of some recent MRI reanalysis projects?

And @ufangyang talked us through how the #EEGManyPipelines project will make progress on the empirical question - they have made huge progress recruiting over 100 analysis teams.
Find out more:
eegmanypipelines.org

So what’s a discussant to do when wrapping up such great presentations of such important projects?
Why look for similarities of course!

Have you noticed we all have red EEGs in our logos 😹

More importantly, all 3 are all motivated by the challenges of analytic flexibility.

In the multiverse analysis shown here just two to three options at eight different processing steps results in 864 different ways to analyse!
So many options we invented a new plot 😅

And although our multiverse analysis reveals that some overall events like this N400 remain significant in all pathways, let’s see what happens when we investigate pairwise comparisons between conditions…

Now we can see there are important differences between. Some comparisons are only significant in some conditions 🤨
Read more?
psyarxiv.com/8rjah/

This has important implications not just for this study…

There are impacts on
👉 Replication. What future studies will replicate?
👉 Metascience. Which data is valid to pool across or compare?
👉 Credibility. Who will believe a single finding in our field?
👉Ethics. Responsibilities to our participants AND to our junior trainees!

🙌🏻Louder for the people in the back!

https://twitter.com/CoSci_EEGpers/status/1575626377063436288

So what can we do?
Historically we have seen Complaints (position/theory papers) and Advice (guidelines/handbooks). The field might look quite different if we mandated best practice — but at the moment, compliance is patchy.

These 3 projects focus on pathways to solutions

Each project tackles analytical flexibility from a different perspective, but all three are community-led projects aiming to make research easier through collective action

And these 3 projects – like some of the best in Open Science – build up diverse, global networks, using pipelines for distributed work and collective decision making

This is the future of neuroscience 🥰

I love all of these projects and they’re making great progress 💪🏻

oops - thread continues here:

https://twitter.com/suzyjstyles/status/1575796004162396160

@threadreaderapp unroll