1/ I asked #GPT4 to review our paper with @LDjaparidze. This is what happened and what I learned in the process. medrxiv.org/content/10.110…
2/ Since the general problem that practitioners find (in the worst way) is always training set tainting (guilty-as-charged). Habits die hard, the first thing I did is asking to do a review of the paper without any extra knowledge about what the paper says
https://twitter.com/TheLeanAcademic/status/1637790306975326208
3/ From the response alone I learned 2 things. First, our paper title was deadly accurate. I also learned that it has no information whatsoever on it, as the entire response can be generated from understanding the title itself. 
4/ Tried a few times to figure out if there was leakage, but couldn't find signs that would prompt me into the paper having been on the training set (or if it is, it is very diluted). Sent the main body text (including tables) and ask to do an "accurate peer review".
5/ #GPT4 is a very capable summarization engine. I
(as the author) would say that the summary is quite good even though it hallucinated the title. There is a very interesting tidbit, it is showing that it is losing context over time, and will be evident in following examples.
(as the author) would say that the summary is quite good even though it hallucinated the title. There is a very interesting tidbit, it is showing that it is losing context over time, and will be evident in following examples.
6/ In the work we introduced an innovative way to constrain the model equations using data that can be collected in field studies. This have very profound implications for the accuracy of the models. So I asked #GPT4 to explain why we did it.
It checks out.
It checks out.
7/ Our paper is 40 pages long. And the second appendix includes everything that we learned after the initial release date (we wanted to keep the body fixed with the data up to when the analysis was performed). So I reset and added also the second appendix.
8/ In here is where things get interesting. It is pretty obvious that the context reaches its limitations. After immediately finishing #GPT4 offers a summary that just focuses on the Appendix 2 findings, not on the main paper.
9/ This becomes much more evident when I asked it to propose an abstract. Though based on this proposal it looks like that there is enough substance to just convert the Appendix 2 into a whole paper :D
10/ This was fun.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
