Thread by @asclnet

First talk on this last day of #ADASS2019 is by André Offringa, who is talking about Designing radio-astronomical software for delivering science-ready products. The abstract for his talk is here: adass2019.nl/abstract/?tabl…

Radio data has large data volumes, requires lots of processing and computing, and requires novel algorithms.
#ADASS2019

#ADASS2019

What is science-ready data?
At least a high quality image; often, more is required to extract science, such as source positions and size, spectral indices, power spectra, and polarization.
#ADASS2019

In ideal situation, astronomer has an idea with a certain hypothesis, requests and gets observing time, receives the "science-ready" data products, is able to immediately answer the hypothesis, and then gets a Nobel. :-D
#ADASS2019

One advantage of this ideal situation is that there is almost no redundant processing knowledge by the astronomer
#ADASS2019

But that's not how radio astronomy traditionally works. Observatories just produce the data, many PhDs are spent on data processing, and many tools are written to solve the same problem. Also, telescopes are only fully accessible to expert teams.
#ADASS2019

LOFAR observatory in 2013 started doing preprocessing of data, and in 2018, included lossy compression. Now it does full direction-independent calibration.
#ADASS2019

Making radio-data processing pipelines is challenging! It's complex, high performance, experimental, AND it's hard to get a grant to "write a generic pipeline" as the common answer is "that's not science."
Also, there is no money/resources/credits/plans for support.
#ADASS2019

The difficulty of writing good software is typically undervalued by management.
(Me: many are trying to change this, and work for better recognition and support for software creation and sustainability.)
#ADASS2019

#ADASS2019

AOFlagger is in the ASCL here: ascl.net/1010.017
#ADASS2019

He wrote a GUI for AOFlagger so users can experiment with the settings, but he didn't know what people were doing with it. So he wrote a Python interface, but it was far too slow, so the old interface is still used.
#ADASS2019

He has also worked on WSClean (ascl.net/1408.023)
#ADASS2019

#ADASS2019

It's hard to combine two existing packages, maintain good performance, and have a good interface.
#ADASS2019

Summary:
Radio processing is challenging
Making observatories produce science-ready at a is of high importance.
An increase in resources for the central development of processing algorithms (inc. maintenance and support) will result in larger science output.
#ADASS2019

Of high importance because:
- much lower learning curve for astronomers
- processing experts at observatories, reuse of code
- science accessible to wider community
- increased science output!
#ADASS2019