Menu
Sign In Search Podcasts Libraries Charts People & Topics Add Podcast API Blog Pricing

Reiner Pope

πŸ‘€ Speaker
1157 total appearances

Appearances Over Time

Podcast Appearances

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

this is really i don't have these numbers memorized but generally as you go to slower tiers uh flash is plausibly in the order of one minute um and then like spinning disk which is massively different i think is on the order of one hour so this might actually identify that the tiers are probably flash and spinning disk sorry why why is this the calculations the storage cap divided by the bandwidth so um you've got a bunch of different memory tiers like we've listed four of them um

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

uh the your choice like your choice of which memory tier is like you want to minimize the cost yeah um and so you are like what fraction of the device are you using you're using some fraction of the device for the holding onto it and then using some fraction of the device to retrieve it um and so let's say

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

I'm using like 10% of the device and I want to equalize those two fractions.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

That's a sign that I've hit the right thing.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

So let's say I've got some runtime here, like I'm going to hold on for all of this time.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

And then, so this is the time hold.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

And then there's going to be some amount of time here, which is time retrieve.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

And I want, I mean, basically to equalize the costs, these two costs, um, I want the retrieval time to be equal to the hold time, uh, times the like fraction of capacity.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

because this is the retrieval time.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

Yeah, I mean, this is how many other things I can hold simultaneously.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

I think that probably indicates that this is the two tiers of Flash and Spinning Disk.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

I'm kind of shocked to see Spinning Disk being used at all because it's such an old technology.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

Yeah.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

It's a really unattractive technology, but it's useful in some places.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

Yeah.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

So, I mean, like, the mixing, like, I try to look for other examples where mixing, like, scrambling mixing shows up as well.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

There's actually almost even, like, a physical example where...

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

Like you're stirring something, you're making a cake and you want to stir the batter.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

And like literally the idea, like first stir it this way and then stir it this way is like actually not too bad of an approach.

Dwarkesh Podcast
Reiner Pope – The math behind how LLMs are trained and served

But beyond that, like back to the digital world, there are some differences.