Illia Polosukhin

I mean, you know, as I talk with researchers, this is probably one of the main challenges that everybody's targeting right now.

Yeah, I mean, effectively, the way to think about it, I mean, we can go physiological where, you know, the human learn, whatever.

In the span of years, yes, you only maybe...

have like 80 million tokens in 10 in a decade right so it's not like you're actually not getting that much like language tokens but you have visual tokens you have like you have physical you have all of this additional information and that actually is like our what what kind of goes from a pre-trained model we are born with right to a

fully, fully fine-tuned, you know, people we are.

And so AI right now, yeah, as I said, like it's a, it's just like genius in the memento, in the memento state, right?

And so to really unshackle it more, you kind of really need to, this,

longer context and like it already has ability to learn in context so this like concept of in context learning right so if you if you show it something it didn't know about before it will start using it but it needs to be in the context and so you know as you show it like here's the thing i want you to do and then you know like it goes does a bunch of stuff all of that is fills its context and now

like, again, all the actions, all the responses, like, if it read an article about, you know, for example, preparing for this interview, it went, read an article from Nir, like, all of that now is in its context, right?

And, like, there's techniques to kind of compress it, summarize it, you know, have sub-agents to do a bunch of stuff.

So there's, like, different ways to, like, mitigate it, but at the end, still, like, at some point, it's like, okay, I'm out of context, and now to do next thinking...

I need to clear stuff up.

I need to remove something.

Yeah.

And at that moment, it's a very lossy because it doesn't actually know what's useful, what's going to be useful in going forward.

Right.

Now, again, there is ways how this is addressable with like a longer term memory.

And this is what, again, what's open claw, I think, I get pioneered is this idea of kind of memory tools.

Like there's been a lot of work on that, but they kind of done like a reasonable approach

set up for that but this is just the beginning right like and and and it's a still pretty fixed tools right it doesn't have some of the semantic linkage of like okay well those things are more relevant than this like for this events for this context etc so anyway there's going to be like massive improvements over this year in in all of this and i think the other interesting thing where i actually on engineering side for example right now like