Dwarkesh Patel

Right.

Yeah.

Okay.

And how does the pie model work?

And like what is actually happening is that it's like predicting I should do X thing, then it's like there's an image token, then some action tokens, like what it actually ends up doing, and then more image, more text description, more action tokens.

Basically, I'm like looking at what stream is going on.

Right.

I find it super interesting that, so I think you're using the open source Gemma model, which is like Google's LLM, the release open source, and then adding the section expert on top.

I find it super interesting that the progress in different areas of AI is just based on not only the same techniques, but literally the same model that you can just use an open source LLM and then add a section expert on top.

It is notable that like you naively might think that, oh, there's a separate area of research which is robotics and there's a separate area of research called LLMs and natural language processing.

And no, it's like it's literally the same.

It's like the considerations are the same.

The architectures are the same.

Even the weights are the same.

I know you do more training on top of these open source models, but that I find super interesting.

Today, I'm here with Mark, who is a senior researcher at Hudson River Trading.

He has prepared for us a big data set of market prices and historical market data.

And we're going to try to figure out what's going on and whether we can predict future prices from historical market data.

Mark, let's dig in.

Happy to do it.