Thread by @tallphil

Here's a twitter version of my @OBF_BOSC #BOSC2020 #BCC2020 talk about @nf_core 🚀

@nf_core is a community effort to collect best-practice analysis pipelines built using @nextflowio. They run on almost any computational infrastructure and bundle all software requirements.
[1/16]

There are @nf_core community guidelines to ensure that all pipelines are consistent and reproducible. To help pipeline users and developers there is a command-line helper tool (github.com/nf-core/tools) and of course we create and maintain a set of pipelines 🧑‍🔧

[2/16]

There are currently 25 pipelines with stable releases. Each release gets a @ZENODO_ORG DOI and tagged @docker
images for ultimate reproducibility.

We aim for one @nf_core pipeline per analysis type, collaborating on adding new features for the whole community.

[3/16]

🆕 Bigger @nextflowio pipelines can have a lot of parameter arguments. @nf_core pipelines will soon all come with a new file describing each parameter in a structured manner 💻

[4/16]

Pipeline parameter descriptions are written using the standardised @jsonschema format and allow us to automate loads of stuff: validation of inputs when running a pipeline; command line help; even the website documentation - all from a single file 📔🪄

[5/16]

Large @jsonschema files are difficult to write by hand, so we made a helper tool 🤖 This scans the pipeline code and builds a schema using the parameters it found. It also checks that the schema doesn't have any unrecognised parameters.

[6/16]

The tool launches a GUI on the @nf_core website. You can group parameters, reorder with drag and drop and write help text using an intuitive interface. You can even choose an icon for each parameter! 🤩

When you're done, the command-line tool saves your updated schema.

[7/16]

One of the most exciting things this new @jsonschema allows us to do is to build user interfaces for launching pipelines 🚀

Some pipelines now show a new "Launch" button on the @nf_core website, eg. nf-co.re/atacseq

[8/16]

Clicking "Launch" takes you to an auto-generated form where you can customise the pipeline parameters for your run. It shows help text and validates your inputs in real-time. 🧙‍♂️

[9/16]

When you're done, the interface gives you a command that you can copy and paste in your terminal to kick off the pipeline run.

You can also use the command-line tool to run this interface with *any* @nextflowio pipeline (if no schema is found it auto-generates one).

[10/16]

If you're running offline you can do all of this interactively on the command line instead! 🖥️⌨️

An interactive wizard takes you through the parameters and even renders the #markdown help text in the terminal using #Rich (github.com/willmcgugan/ri…)

[11/16]

Next up! 🆕 We have a pipeline code linter tool to check @nf_core guidelines. It runs using @github actions, but results can get lost in the logs

Soon, ⚠️ and ❌ will be automatically posted as comments on PRs, with descriptions and links to the documentation.

[12/16]

We do test pipeline runs using @github actions too, but with tiny datasets due to limited resources.

We wanted to run more realistic full-size datasets, but we were going to need something bigger..

[13/16]

Thanks to a grant from @awscloud we are now running automated analysis with full-size datasets for every pipeline release. These use #awsbatch and we can monitor the runs using @SeqeraLabs @nextflowio Tower.

[14/16]

This year @nf_core was published in @NatureBiotech 🎉 nature.com/articles/s4158… (full text: rdcu.be/b1GjZ)

You can see live versions of the plots from the #supplementaries on the @nf_core website: nf-co.re/stats

[15/16]

The @nf_core community continues to go from strength to strength - we have hundreds of contributors and a large and friendly @SlackHQ organisation, with dedicated channels for each pipeline.

Come and join us! nf-co.re/join 👋

[16/16]

If you prefer, you can watch the recorded version of this talk on YouTube:

Feel free to post any questions on this twitter thread, or come and join us in #Slack - nf-co.re/join