Thread by @RoyKishony

Introducing "data-to-paper": autonomous AI research! We've let it play with the large CDC Health Survey Dataset. Went to lunch. When back, it had already chosen several research topics, wrote data analysis codes, interpreted results and wrote 5 transparent, reproducible papers.

Works by creating interactions among ChatGPT and algorithmic agents that take on different roles (including “scientist”, “reviewer”, “coder”, “lit reviewer”, etc). These agents are automatically guided through the canonical sequence of research stages; from data to paper.

Here is an example manuscript that data-to-paper produced autonomously. The only thing we gave it is a description of the data. repo / web coming soon, stay tuned. With the amazing @TalIfargan.
drive.google.com/uc?export=down…

@berkley_lynch by "reproducible" we mean that a produced paper includes a well-structured Methods section explaining the study's process, and that all the data analysis codes written by chatgpt for the study, + output of these codes, are provided with the paper. drive.google.com/uc?export=down…

@berkley_lynch All references are genuine. We don't let chatgpt invent citations, rather allow it means to search for citations. Admittedly, it is not always the best choice of references, but they are real and with reasonable fit. We will improve more on that front.

@AltenbergLee Perhaps there is an opportunity with such tools to better formalize multiple hypotheses in exploratory science. This is something that is currently very hard to do. If we use this system adequately (and honestly), it can be a way to systematically formalize such corrections.

@AltenbergLee surely this is a good point!
in general, even without such automation, data science research has become so easy and systematic that I would have voted for reducing the traditional 0.05 magic value to a much smaller number.

2. Get you thinking of what type of questions and problems we human scientists should focus our minds on in this new era. Hint: it won't be on simple straightforward analysis of data.

@danofer yes it is far from perfect. and yes chatgpt does have hallucination problems. with all of that, when given a series of small tasks with tight instructions and feedback (automated) it can create a solid scientific paper. drive.google.com/uc?export=down…

@aldertzomer We gave it the raw data, from Kaggle.

It is a cleaned version of the CSC's BRFSS data. unweighted.kaggle.com/code/alextebou…

@rmn @Teknium1 Codes and outputs in the supplementary of the produced paper.
it is not perfect for sure. eg in terms of which confounding factors, etc. but the point is that it gets you thinking of what ChatGPT can do.
drive.google.com/uc?export=down…

@BlancheMinerva @berkley_lynch you are right!
it is reproducible in the "technical sense": Our aim is that the produced papers do not just give an end result, but also describe how it got there, so that *human scientists* can understand and technically/practically reproduce the different steps.

Just to clarify: “reproducible” just means “reproducible in the technical sense”: created papers describe not only an end result, but also the *technical* steps and the analysis codes used by ChatGPT to get there, so that *human scientists* can follow, review, and reproduce.

Here are n points I would make, on how we view data-to-paper:

Here are n points I would make, on how we view data-to-paper:

0/n. data-to-paper is just a platform that runs the “good old” ChatGPT. It is not a new language model. Similar to @Auto_GPT, @LangChainAI, others, it calls ChatGPT in a structured automated way with multiple steps towards a complex goal - here analyzing data and writing a paper.

1/n. Primarily, it is meant to demo the power of ChatGPT in Science, an area often considered a castle of human ingenuity, thus provoking discussion on the super-strengths and society impact capacities of even current LLM models (and surely upcoming ones).

2/n. It get you thinking of what type of questions and problems we human scientists should focus our minds on in this new era. Hint: it is not going to be simple straightforward data analysis.

3/n. We want to think about how we might use such systems more interactively, like as a "scientist co-pilot" that does the more straight-forward parts of science, allowing us to work at higher levels of abstraction and focus on higher-level questions and challenges.

4/n. ChatGPT can hallucinate. Hallucinated citations are solved by access to search engines. Other hallucinations are minimized with auto-checks, with mutual ChatGPTs, and by providing well-defined tasks. Yet, for sure, high-quality papers require human judgment and assessment.

5/n. Created papers are not perfect. In particular, the statistical tests done by chatgpt can have issues, e.g.: not always correctly controlling for confounding variables. We would love specific comments on strengths and weaknesses of the example paper: .drive.google.com/uc?export=down…

6/n We are limited to papers that perform statistical tests on a dataset. While we might be able to extend to some other types of data analysis studies, there are many other very different types of studies that are not at all possible with current LLMs.