Thread by @jaschasd

Title: Advice for a young investigator in the first and last days of the Anthropocene

Abstract: Within just a few years, it is likely that we will create AI systems that outperform the best humans on all intellectual tasks. This will have implications for your research and career! I will give practical advice, and concrete criteria to consider, when choosing research projects, and making professional decisions, in these last few years before AGI.

This is my current go-to academic talk. It's mostly targeted at early career scientists. It gets diverse and strong reactions. Let's try it here. Posting slides with speaker notes...

--

The title is a play on a very opinionated and pragmatic book by the nobel prize winner ramon y cajal, who is one of the founders of modern neuroscience.

To get you in the right mindset, on the right we have a plot of GDP vs time.
That is you, standing precariously on the top of that curve.
You are thinking to yourself -- I live in a pretty normal world.
Some things are going to change, but the future is going to look mostly like a linear extrapolation of the present.

And the plot should suggest that this may not be the right perspective on the future.

This plot by the way looks surprisingly similar even if you plot it on a log scale. We didn't stabilize on our current rate of growth until around 1950.

No notes, just a talk outline.

Let's talk about geologic epochs.

They are a measure of deep time, divided by distinct changes in life forms, climate, and geological processes, that can be observed in rock strata.

Humans are starting to have geologic impact. We are causing the 6th mass extinction in the history of the earth. Radioisotopes from nuclear weapons tests are being captured in rock strata. You will be able to tell, a billion years in the future, that we were here.

There's a proposal to name this current period of human driven global change the Anthropocene epoch.

This focus on human-driven change is interesting, because we are perhaps near the end of the period of humans being the primary intellects driving global change.

The central conceit of the talk is that, very much depending on the way in which AI plays out, this could be a very, very short geologic epoch.

Here is a plot of the amount of compute used to train notable AI models, starting at the beginning of the Anthropocene, through the present. You can see a change in slope around the point where AI models became economically valuable.

Each tick on the y axis is two orders of magnitude larger than the one below it.

In the upper right I've circled a rough plausible range for the compute performed by the human brain in a lifetime.

My own back of the envelope estimate, assuming one floating point operation per synapse per millisecond, is 1 trillion petaflops, or a fraction of a grid division larger than current models.

So we are reaching scales of model training compute similar to those in the human brain across a lifetime.

Though not necessary or sufficient, approaching human levels of compute makes it more plausible that we are approaching human levels of intelligence

It's not just the scale of AI model compute that has been increasing.

Here's a recent study from METR looking at the timescale of diverse software tasks which AI models can perform unaided, with 50% success. The timescale is derived by looking at how long it takes humans to perform the same tasks.

here on the x-axis we have the year the model was produced. Here on the y-axis we have the task length that the model can succeed at with 50% probability.

The key thing to observe is that it is increasing exponentially. If we extrapolate, we would predict that models can perform a full day's work (at least in software), unassisted, with 50% success, sometime in 2027.

The curves for higher success rate have the same slope, just a different intercept. So if you want 80% success, you should add a year or so to your AGI timeline.

And here's a plot showing the speed at which superhuman performance is achieved on new benchmarks, up through 2023. Up through two years ago. Superhuman performance is reliably achieved, and it is reliably achieved faster and faster.

Here the y-axis is performance relative to a human baseline. You can see that models were not superhuman on benchmarks that were released in 1998 until around 2015, while superhuman performance was achieved on a reasoning benchmark introduced in 2019 by 2023.

I was one of the organizers of a very large scale collaborative benchmark called BIG-bench. We had publications reporting superhuman performance on subsets of it before the paper announcing the benchmark was even published.

--

This trend of rapid solving of ever harder benchmarks has continued since 2023. Here is progress on GPQA Diamond, which was a benchmark created by PhDs in different STEM fields, in a 5 stage process of iterated creation, review, and filtering.

You can see that it took a year and a half after this benchmark was introduced, for AI to go from barely above random chance performance -- that's the bottom dashed line -- to superhuman performance.

Expert level here -- the top dashed line at 70% -- is the performance achieved by PhD students in the particular scientific field that the question probes.

The first time I gave this talk, [famous AI researcher] was in the audience, and insisted that improvements on benchmarks were driven by benchmark questions being added to the training set.

Here is an experiment directly addressing this. A group of researchers at Scale created de novo questions, carefully matching the core skills and difficulties probed in an existing benchmark of math word problems.

Plotted is the relative performance of LLMs on the new questions, compared to on the original benchmark.

A y-axis of 0 would mean they did just as well on the new questions as the original benchmark questions. A y-axis of less than 0 means that the model did better on the original benchmark. That is, a y-axis of less than 0 suggests that the model was cheating, and was training on the test set. From this, you can see that there are labs that are training on the test set!

If you look closely though, all the labs that are cheating are the ones that are behind and struggling to catch up.

The frontier labs of Anthropic and OpenAI and mostly Google all do roughly as well on the new questions as they did on the old questions.

I'm particularly proud that Claude actually did even better on the new questions than the original benchmark. We are not overfitting to this benchmark.

--

And it's not just academic benchmarks. Earlier this year, models from both Google and OpenAI achieved Gold Medal performance on the International Math Olympiad.

The IMO is a high school age math competition, that involves novel problems every year ... and it is really very, very hard. If you are not familiar with it, and don't believe it's hard, I suggest you try to do some of the problems, or ask a friend in the hard sciences about it. This year, in the whole world, 72 students achieved gold medals ... and two LLMs did.

Because these problems are made just for the competition, if an LLM gets them correct, it is not due to overfitting.

Here's a quote from the competition organizer: "[AI] solutions were astonishing in many respects. IMO graders found them to be clear, precise and most of them easy to follow."

From conversations with people at multiple other labs, I think we're likely to see AI solving real open research problems in math in the next months. This isn't something I can show you yet -- but I would be happy to make bets on whether there will be published examples of models solving math problems no human has yet solved within the year.

When I gave this talk at Harvard, I got this fascinating unsolicited email, where the entire message was "What a bat-shit crazy abstract...".

There's this concept called the Overton window. The Overton window is the set of ideas that are considered acceptable to express or publicly hold.

At least on the surface, this email is a statement that AGI is outside the overton window. Or at least that it's inside the Overton window to call predictions of AGI crazy.

Since the Overton window determines what we feel comfortable discussing, and even what we feel comfortable thinking, it's important to communicate that AGI is inside the overton window.

A primary purpose of this talk is to give you permission to take this stuff seriously.

What I am saying in this talk is normal. Respected and well known people and institutions say the same things, and they're still respected.

To illustrate this, let me read some quotes from respected people predicting disruption from AGI. [...]

And it's not just individuals. Journalists are flirting with the idea of AGI too. There are stories and opinion page pieces in serious newspapers taking the idea seriously.

If these people take the idea seriously, you have permission to take it seriously too!

So models are getting better very fast. And I've set an ominous or promising tone by talking about the start and end of a geologic epoch. I've talked about how the idea of AGI is normal enough now, that you're allowed to believe in it in polite company. But when can we expect it to happen?

When can we expect models to be human or superhuman on different things? Especially when can we expect them to be able to automate the kinds of tasks that make up the majority of our work?

If you live in the SF AI bubble I live in, the consensus answer is just a few years. People describe themselves as having long timelines if they think it will take a decade.

This is unusual, because practitioners are the most aware of the practical barriers that limit progress in the short run. My partner is in biotech, and if you ask researchers in biotech about exponentially declining cancer rates, the first thing they do is give a long list of barriers to eradicating cancer. (and cancer rates are exponentially dropping -- it's amazing! optional slide on this later)

So when experts are saying there are no blocking near term barriers to AGI, that is something to take seriously.

Here are surveys of contributors to major AI conferences.
The y-axis is the probability that ML paper authors gave to reaching AGI, or human-level-machine-intelligence, by each year on the x-axis. Each light line in the background is a single random participant.
In 2022 the population gave a median estimate of AGI by 2060 or so.
In 2023, the median estimate was 2045 or so, with many individuals predicting much shorter timelines.
These surveys are now two years old. I don't know what a survey taken today would show, but I'm betting on a median estimate in the 2030s.

I can also speak to my personal experience. I've mentored a lot of grad students. Currently Claude feels like working with an inept grad student that gets silly things wrong, but also one with encyclopedic knowledge of every field, who is eager and amazingly fast.

This description wasn't true one year ago. It won't be true again one year in the future. The models will be much better than they are today.

optional slide -- depending on whether audience ornery about definition of AGI

It's common in conversations about AGI, for someone to get stuck on what the definition of AGI even is. There is a lot of nuance involved in finding a definition which is both unambiguous and satisfying!! I've published a paper on this.

For most purposes though, it is a mostly irrelevant question. There will be a period (right now) where we can have long arguments over whether machines can think. Then immediately after it there will be a period where the question seems silly. When the answer to whether a machine can do a given cognitive task better than a person is "of course", then we will have AGI.

If we take these timelines seriously, what should we as individuals do??

These are plausibly our last moments to intellectually shine.

First, we should make sure our projects will still be relevant when they are completed. There is some improving curve of the science that can be done with minimal human effort, just by prompting the best current foundation model. One outcome that you want to avoid is that you work hard for 2 years, and make significant progress, but by the time you're done someone can achieve something better just by asking a foundation model.

This suggests working on targeted projects, with others, so that you can go fast, and stay ahead of the exponential. It discourages slow open-ended exploration.

In 2019 Richard Sutton published an essay called the bitter lesson. He observed that general methods that leverage computation are ultimately the most effective.
The bitter lesson has maybe a positive and negative takeaway. if you work on projects that become more effective as compute and intelligence scales, or even better feed back into that process, then the results will be higher impact than you anticipate.
The negative is that you really, really don't want to work on projects that are going to be solved anyway with scale, possibly before you even finish the project.

You should force yourself to use the AI tools available to you.

They provide fundamental new affordances. They are often awkward and non-ergonomic. They are definitely hard to learn to use well. They are an unsolved UI problem.

You should use them anyway. Both because they're already useful, and because it will prepare you for the next generation of tools.

you should use them to [brainstorm, ...]

Working with LLMs is like being a PI... Both activities are great practice for each other. Require clear well defined problems and tasks, of appropriate scope.

This talk was motivated by a recent conversation I had with a very capable grad student, where they were sharing a personal timeline for AGI of 3 years ... and then a few minutes later were talking about their career plans, and ["..."]

If you take seriously that AGI is coming in one part of your brain, you should also take it seriously in other parts of your brain.

I don't know what an academic job will look like in a few decades. For that matter, I don't know what any job will look like. But I am sure it will look quite different than it does today. You should not be planning with an assumption that your current career path is a stable one.

People often make life decisions balancing risk vs. reward. If AI is going to cause large scale disruption, and is going to be able to do your job ... then there is a high baseline level of unavoidable risk. This means that if you are choosing between a safer research or career choice, and one with a higher potential upside ... the safer one probably isn't actually safe, and you are mostly just sacrificing upside.

Even without considering AGI, people are often too risk adverse. Do the interesting thing instead of the safe thing.

if you have a limited time to contribute, and the stakes are high,

you should do something you are proud of!!!

When you are retired in your villa in the Dyson swarm, you will want to feel that you helped get to a positive outcome.

You should also choose projects that will have mattered, looking back. This means you should prefer projects that shift the trajectory post AGI, rather than induce a transient change in the near term.

The photo on the right is the big reveal -- the geologic epoch after the Anthropocene has all rock strata converted into a dyson swarm

On that last point, of doing something that you are proud of

Sometimes people feel a lack of agency over the big picture trajectory.

The progress of the field of AI can seem like a process beyond our individual control.

I'd like to take a moment to emphasize that that perspective is completely wrong.

Despite dramatic capabilities, we are still early in the AI exponential. To the right you can see an extrapolation by Epoch AI to compute in 2030.

The important aspect of this image is that compute has the potential to be 10000 times larger in 5 years. Looking back, our state today will be visually indistinguishable from zero on a linear-axis compute plot.

We are still early in terms of the resources being spent on AI

--

So we're still in the early part of the AI exponential. Why does that matter? Because small interventions early in exponential growth have huge consequences

There are countless examples in science and engineering and governement and economies where small decisions made early in the process led to dramatically different outcomes.

Because we are early in the exponential, and because you are almost certainly capable with highly relevant skills, you have immense power and leverage over the future of AI.

This is a responsibility as well as a power. More than at any other point in your life, decisions you make now are likely to have enormous indirect consequences for large numbers of people you care about. Be intentional and thoughtful about the projects and jobs you choose.

I've told you that I think AGI is coming, soon.
And that I think this should motivate you as you make personal and professional and research decisions.
And I've talked about a few specific ways it might feed into that decision making process.

Now here are some particular project areas I like. This list especially is massively incomplete. The most impactful things to do are probably the ones I've left off, and no one is yet thinking about.

If you want to talk in more detail about what work in any of them would look like, please approach me after the talk.

[read list]

AI for science is things like material discovery, protein folding, weather modeling, fusion reactor plasma monitoring, ...

Science on AI models means treating the AI models themselves as the object of study, and using techniques from another field. Better understanding of the systems we are building is also often a positive.
interpretability is maybe the canonical example of this
there is work treating the parameters or activations of neural networks as stat-mech like ensembles
there is work probing the apparanet psychology of AI models
there is work treating the economic behavior and consequences of models
etc. It might be useful to spend an hour brainstorming all the fields that could be used to do science on AI models

There is straight up AI safety research.
This is something you can get in on the ground floor of, which is hugely important.

There's prediction and extrapolation of AI capabilities. The better we understand what the future might look like, the better it will likely go.

There's access, equity, fairness.
If we want this technology to benefit everyone, this is incredibly useful. I also expect it to be a toxic and stressful subfield to work in. It's embracing multiple culture war issues at once, and your work is only going to be net positive if you are able to do that without being dgomatic.

If you are equipped technically and in personality to do work on policy and governance -- DO IT! Governments are desperate for competent technical people to advise them. Everyone good leaves for more pay. But we really need technically competent people informing these policy decisions. This is amazingly high leverage if you do it.

So in summary -- take the future seriously!

What you work on, where you work, when you make career transitions, how you think about the important and interesting problems, how you think about the potential consequences and leverage of your work, ... All of these are hugely important

The next few years are also a good time, possibly the last time, to go all in. The potential impact of your work is unlikely to ever be larger.

Should I keep on going? The talk is over. But I have some extra slides that almost made the cut... Including a rubric for evaluating research projects.

I promised I would give concrete advice -- so let me share with you an actual rubric I use when choosing research projects. This is slightly off-theme for the rest of the talk. It is partially informed by AGI -- but is a good set of questions to ask about a project regardless of AGI timelines.

First, you should gate by impact. If this project works flawlessly, how large is the potential benefit?
Make sure you evaluate the expected outcome projected onto your value axis, rather than just measuring its norm. See the previous section -- you want to have an impact you can be proud of, and this matters more and more as the potential scale of the impact increases.
This doesn't mean that every project needs to be trying to get a nobel prize.
It's fine to do small projects, but they should be in pursuit of a coherent larger mission or goal. You should be able to tell a story you believe about why they're important.
In fact, doing an ambitious project without breaking it into small steps is a failure mode. If you are going to succeed, you need to be able to break your ambitious project into small quantifiable steps.

In a fast changing field, you should evaluate the impact the project will have when it is complete, not the impact it would have if it appeared now.

--

Bitter lesson: will your research be robust to increased scale of compute, and to increased scale of intelligence! Things that are robust are things like
developing foundational datasets or running experiments that are not easily repeated
developing algorithms that interact superlinearly with scale. you want your approach to work better the more capable its complements are
work that sets the questions that future research asks or the framing that future research uses

--

Opportunity cost: How much time and effort will this project take, that will not be useful if the project fails? It is good to find intermediate steps that will generalize to other projects. It is also very, very, very good to find ways to de-risk toy versions of an idea as quickly as possible, so you waste as little effort as possible on dead ends.

If a project does fail early, it is often still worth the time to get it to a checkpoint you can interpret later, or share with others. That way, when you later discover it was actually relevant, you can go back and pull it back up.

--

Comparative advantage: why are you, in particular, unusually suited to this project? This could be access to data, or compute, or skills and knowledge that you have, or collaborators with specific skill sets, or a clever idea that you think no one else has thought of.

Be suspicious of thinking you have a new insight though! Ideas are rarely as unique as you think.

Redundancy: How many other people on the planet are presently trying to solve the same problem in roughly the same way? Even if you win the publication race, it is wasted effort if the same thing would have been discovered at roughly the same time with or without your work.
You want the world to be in a different state because of your efforts.

This is the one that I see people fail at the most often. If everyone agrees that you are working on a very important problem, then you shouldn't be working on it. Someone else will do it. You want to get on the wave before it's crashing, not afterwards.

This is also the one that applies differently in industry than in research. If you're running a company, often you need to try things that other companies are also doing.

--

Also, I just want to say -- people never go weird enough!!!

Whatever project you are working on, you should choose a weirder one.

You are going to be judged on the best things you accomplish, not on the typical thing you accomplish.

This means you should be optimizing for the long tail of surprising very positive outcomes.

Try something that is weird, but might be amazing, rather than something that is probably OK.

The ideal thing to work on is something where you can clearly explain why it is a good idea, but when you explain it to other people they look at you funny and have trouble getting it. This is the strongest signal of future project success!! You need the first part too though -- you need to at least be able to clearly and concretely explain to yourself why it is a good idea.

So that was a rubric for how to choose an academic research project.

There is a second question, which is hinted at in this talk, which is should you do academic research? There is not a right answer to this, but it is worth thinking through in a structured way.

I can share where I landed on this question, which is that I have mostly stopped doing academic research.
I especially miss being able to tell people about my work.
But there are benefits -- I have more compute, and money, and access to different cutting edge problems, than I think I would have in academia. I also arguably have more leverage on the future.

In a parallel universe, I am running a small academic lab, and loving it. In many ways that is a more appealing life path to me. But I believe we are building a technology that will utterly change the world, and I can't stay on the sidelines for that.

This is a plot from another talk by a colleague of mine, about how the horse population changed after cars were introduced. Obviously a completely different topic...

Let's end on a high note! Here is a followup to the earlier slide, where I mentioned that cancer rates were exponentially falling. Doesn't this image make you happy? (less happy that we need to filter by rich countries. but if we keep making it easier to beat cancer, the rest of the world will catch up)

Actual slides here. Feel free to reuse them however you like.



(did you know that you can't save drafts of tweet threads? this website...)docs.google.com/presentation/d…