Dylan Patel

Humans, maybe we have like a short-term memory and a long-term memory.

I think it's a lot more blurry than that.

There's no like clear line.

Oh, this is in my short-term memory.

Oh, this is in my long-term memory.

It's like, it is much more blurry as we go back and back and back.

It's more and more sparse, right?

And if we think about, hey, what do you remember as a kid?

The most crazy thing in psychology, I remember when I learned it, I was like, wait, my memory of what I did as a kid with my dad at this like thing, right, is fake.

It's me remembering it and inventing the picture and me remembering that picture like successively.

But like the actual memory of what happened is like morphed a little bit over time.

The way humans collapse information is super, super dense, but we were able to extract all the relevant information out.

Now models, there's a ton of research going on in this domain of long context.

How do I get longer and longer context without blowing up my model cost?

This is a big challenge with reasoning.

This is why we had this HBM bullish pitch for a while, right?

It's like, you need a lot of memory when you extend the context.

Simple thesis.

The fundamental algorithm needs to change and improve over time iteratively to get to something like this short and long context of memory.

That doesn't necessarily mean the model has to work like we do.