Thread by @RemiGau

So here is my #OHBM2020Posters 'thread' for #OHBM2020

eCOBIDAS: a webapp checklist to improve neuroimaging methods & results reporting.

The app: ohbm.github.io/cobidas/#/
Poster: osf.io/wt8fj/
Chat: mattermost.brainhack.org/brainhack/chan…
Video chat: meet.jit.si/cobidas_checkl…

#WIP

OK so here is the rest of the thread but before we dive in here are the places where a lot of this information is centralized.

osf.io/anvqy/
github.com/Remi-Gau/COBID…

OK now let’s do this...

Have you ever been in one of the following situations?

1) Writing your first neuroimaging paper but unsure what level of detail to put in your methods section.

2) Reviewing a paper and being unsure what information is missing in that paper.

3) Wanting to replicate some study, reading the paper and finding there is not enough information in the methods section to do that or even to actually make sense of the results.

4) Comparing your results to those of a paper, but not finding enough information about how the authors got there. So you can’t assess where the differences between your numbers and theirs are coming from.

5) Having to look up into your MRI dataset acquisition parameters to write up part of your methods.
And then having to repeat yourself by copy-pasting parts of your methods section when filling in a journal checklist or uploading your results on @VaultNeuro.

6) Doing a meta-analysis and having to design a system to list the methods section of each study

7) Wanting to create pre-registration but not knowing what to put in it.

#ReinventTheWheel

Well we hope that this app can help with a lot of those in the long term.

The fact that we have a chronic problem in methods reporting quality is not new. In a 2012 paper Joshua Carp found that the methods of many papers would not allow independent investigators to fully reproduce their results.

ncbi.nlm.nih.gov/pubmed/22796459

Hoping to help with this problem, the COBIDAS committee of @OHBM was created. It extended previous guidelines to create a report listing what should actually go into a methods section.

COBIDAS: humanbrainmapping.org/i4a/pages/inde…

The report: biorxiv.org/content/10.110…

These guidelines are a great resource and they come with a very useful checklist in the form of a table at the end. The only drawback: it is a 30 PAGES LONG checklist...

And many items only concern some very specific use case ("why do I have to flip through the 2 pages for fMRI if I do tractography?")

So... Some of us decided to make this more user friendly. This was started at the @OhbmOpen hackathon in Rome in 2019. We got together and started breaking down those 30 pages into a spreadsheet.

@cassgvp had already started a lot of the work so that definitely helped.
osf.io/ery2q/

So now those 30 pages have been turned into a "much more manageable" checklist spreadsheet with... 462 items (and growing).

If you want to get dizzy you can have a look at it here:
docs.google.com/spreadsheets/d…

How is that any better? Well I am glad you asked!

This format now becomes machine-readable. We can feed that into some app to render a checklist. And that brings us to the other aspect of that project. How to visualize this checklist?

We have been using some of the work done by the @ReproNim team to standardize how questionnaires are “represented” across studies in a project called Reproschema.

github.com/ReproNim/repro…

This project aims to create a standardized library of questionnaires using linked-data and has a user interface to display those questionnaires and collect data.

And that’s pretty much all we were looking for to get that checklist running.

ohbm.github.io/cobidas/#/

At the moment the app is just a proof of concept based on the 90 metadata items one can fill in when uploading their neuroimaging results to @VaultNeuro. But we have been working on extending that to 200 items from our mother of all spreadsheets very soon.

You can click your way through the different sections of the checklist. There is a bare minimum of input validation. The app tries to minimize which items you are “exposed” to. You can export the results in a very simple text file that encapsulates “all” of your method section.

The next steps (beyond expanding the number of items)?

1) Using that simple machine-readable text file to automatically write large chunks of the method section--like fMRIprep does for you, depending on the options you chose to run it.

fmriprep.org/en/stable/citi…

2) Minimize as much as possible the time it takes to fill in the checklist by making sure that users don’t have to scroll through irrelevant items and by allowing the app to read from a @BIDSstandard dataset to fill in automatically many aspects of the list.

3) The same could be done for some points of the “analysis section” of the checklist by reading info from a NIDM-results zip file.
github.com/incf-nidash/ni…

4) Expand to the checklist to cover EEG and MEG work

@aina_puce @CyrilRPernet have been leading a similar effort to create guidelines for electrophysiology studies.

osf.io/a8dhx/
cobidasmeeg.wordpress.com

And @fedeadolfi @martinagvilas already turned a lot of those into another “tiny” spreadsheet.

docs.google.com/spreadsheets/d…

So there is a good chance that the app could cover EEG and MEG studies too.

Important thing to mention: all of this is a community based effort where most of the progress was done during hackathons @OhbmOpen 2019, @improvingpsych
2019 and @openmrbenelux 2020.

github.com/ohbm/hackathon…
osf.io/har94/
github.com/OpenMRBenelux/…

So a special thanks to all the people who have so much time looking at spreadsheets that they ended up crying tears of blood.

@cassgvp @IRuotsa @fedeadolfi @akeshavan_ @Tim_van_Mourik @davidwmoreau @ZSjoerds @angietepp @martinagvilas @k_wiebels @johalgermissen @DorienHuijser

@WeedaWouter @etklapwijk @RuudHortensius @JamesEBartlett

Contributors list:
github.com/Remi-Gau/COBID…

A special thanks to @ten_photos for connecting me with the amazing work done by Sanu and @satra_ on the Reproschema @ReproNim

And in case you want more methods checklists, you might want to check:

1) The transparency checklist @BalazsAczel
shinyapps.org/apps/Transpare…

2) The CLAIM checklist for AI in medical imaging @neuroccino
claim.shinyapps.io/CLAIM/

3) The CREDnf checklist for cognitive-beharioural neurofeedback studies @rt_thibault
crednf.shinyapps.io/CREDnf/