Ben Thompson
speaker
42,103 appearances
120 recordings
2 series
first heard Sep 2024
last heard 21 Aug
Ben Thompson’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 39 in all, peaking in Mar 2026 with 5.
Appearances
Tens of thousands of things.
podcast-host
And it's gonna be like hundreds of thousands of them.
podcast-host
And and so this is a
podcast-host
There's a lot about the way development for AI has gone that has been very focused on this problem.
podcast-host
How do we execute stuff quickly, but then keep the executors full so they're being utilized all the time?
podcast-host
And
podcast-host
It turns out
podcast-host
Because they're GPUs and GPUs are fairly flexible, not as flexible as a CPU, but more flexible than like an ASIC.
podcast-host
This is also an architecture that works for inference, right?
podcast-host
What do we have?
podcast-host
What's the inference problem?
podcast-host
You need to get the model into memory, and then you also need to house this KV cache.
podcast-host
So if you have all these GPUs linked together, you can solve both problems.
podcast-host
You can get large models into
podcast-host
Like a pod, and you can have the KV cash issue.
podcast-host
Yeah.
podcast-host
But as it gets larger and larger, actually, the KV cache is is getting to be a problem even for GPUs.
podcast-host
So, like NVIDIA's announced this entire like their own thing called dynamo for inference in general, but they've announced like a whole rack for their systems that's just memory, it's just SSDs, and the whole point of that memory is for KV cash.
podcast-host
But but even then, like
podcast-host
If you fast forward five years, fewer, I don't know, but you have these agents where again, you sort of there's no limit to how many agents you might want.
podcast-host
Showing 3461–3480 of 42,103 · page 174 of 2106
← Previous
Next →