Gave a talk today to @cepr_org thanks invite from @akorinek ; great discussion by @Afinetheorem here are my slides in PNG form b/c of the link this 1/N
• • •
Missing some Tweet in this thread? You can try to
force a refresh
Normally, it's: 1) write a paper & submit 3) get reviews (~3 months) 4) revise paper & resubmit 5) wait for response (~3 months)
...what if we could simulate this process in minutes?
Could we fix issues?
Anticipate misconceptions?
Get ideas for new analyses/experiments?
1/
To do this, I created a Python notebook (link at the end) that simulates the process. It lets you upload a PDF & simulate steps 3, 4 + 5 (and so can you, with your own paper):
I created referee personas: 2/
I define the review instructions and & data needed at each step (the paper itself, the review, the response to reviews, etc.): 3/
The "there should be a rule/law/training/office" impetus after every bad event is both a) understandable and b) how you get the kinds of organizations most people hate to be in
For the specific thing I'm sub-tweeting - who exactly do ppl think would provide the oversight that would prevent this? Almost all the peculiarities of research orgs are a reaction to how difficulty the assessment of research is in practice; some putative office of research non-fakery staffed w/ non-experts faces the same problem but w/o a chance of solving it
TBC, I'm not diminishing how bad this is---it's terrible & unlike a lot of research fraud we see, it actually mattered---I'm sure it changed how many people were thinking, making career decisions, making investment decisions and so on.
I wanted to do a thread about how I've been thinking about Gen AI & production. So, imagine a job that's a sequence of tasks to be done 1/n
One of these tasks *might* be doable with AI, so we give a shot by asking nicely:
It will almost always give a shot ('let me delve into that for you...") but then you have to evaluate the output to decide if it is satisfactory for your particular purpose
I'll do a longer thread---and this little toy example just scratches the surface of what's possible---but edsl (pip install edsl) is an open source python package & domain specific language for asking LLMs questions
You can ask questions of various types (free text, multiple choice, numerical), combine them into surveys, add skip logic, parameterize w/ data (to create data labeling coding flows) & then administer to or model models. It abstracts away from the model-specific details
And let's you express the job you want to do in a declarative rather that procedural way e.g., I want to run this survey, with these scenarios, with these agents, with these models etc. rather than writing out all these loops yourself.