Cal Newport

This was the so-called deep learning revolution, really picked up speed in the 1990s into the 2000s, and it was critical for what we have today.

The second big idea that led to our current AI world was Google research notion of adding something known as an attention transformer.

It's a mathematical formalism you add into the feedforward architecture of a large language model that allows there to be a selective attention on the input tokens that

as you are trying to calculate what token to produce next.

This allowed sort of coherent generation of text based off a very complicated input or request, right?

This was revolutionary.

This is largely a sort of intuition around architecture and linear algebra.

The third

Breakthrough that led to our current moment was researchers at OpenAI, led by Jared Kaplan, wanting to know what happened if you scaled the size and training compute of large language models well beyond the limits where traditional machine learning theory said you're going to start overfitting.

Those three things together led us to this current future, and none of those had anything to do with computer programming.

It wasn't, oh, if only we could have programmed faster or been better at finding bugs in computer programs, we would have had those advances quicker.

These advances were scientific, not engineering.

AI does not advance.

It's not created at the fingertips of computer programmers and speeding up those computer programmers does not speed up the rate at which we get smarter, more capable, or more advanced AI systems.

All right, the second point I want to make here

is that these tools, these software development tools that they're testing in these charts, are completely controllable.

Arguments that say we're on track to losing control of AI like to think about these sort of AI-based coding tools as some sort of...

unknowable alien black box that we asked to do things.

And we don't know what it does.

It just kind of like goes off and behave.