Scott Alexander

Maybe if they make it even bigger still, they'll notice more of these connections.

And then third thing, and here's I think the special one, have you tried training the model to do the thing?

Just because the pre-training model

the pre-training process doesn't strongly incentivize this type of connection-making, right?

In general, I think it's a helpful heuristic that I use to ask the question of, like, remind oneself, what was the AI trained to do?

What was its training environment like?

And if you're wondering why hasn't the AI done this, ask yourself, like, did the training environment train it to do this?

And often the answer is no, and often I think that's a good explanation for why the AI is not good at it, is that, like, it wasn't trained to do it.

Like, wouldn't it be really gnarly to try to set up an RL environment to train to make new scientific discoveries?

Well, so, I mean, in our scenario, they don't just, like, leap from where we are now to solving this problem.

They don't.

Instead, they just iteratively improve the coding agents until they've basically got coding solved.

But even still, their coding agents are not able to do some of this stuff.

Like that's what early 2020s, like the first half of 2027 in our story is basically they've got these awesome automated coders, but they still lack research taste and they still lack maybe like organizational skills and stuff.

And so they need to like overcome those remaining bottlenecks and gaps in order to completely automate the research cycle.

But they're able to overcome those gaps faster than they normally would because the coding agents are doing all the grunt work really fast for them.

I'm glad you mentioned that.

Our current scenario does not really take that into account very much.

So that's an example in which our scenario is possibly underestimating the rate of progress.

We're trying to be sort of like our median guess.