Rob Wachen

speaker

529 appearances 1 recordings 1 series first heard Jun 2026 last heard 30 Jun

Rob Wachen’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 1 in all, peaking in Jun 2026 with 1.

Appearances

newest first · ▶ plays the moment

Invest Like the Best with Patrick O'Shaughnessy · Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480] · 30 Jun 2026 podcast

So if you want to serve a giant scale, a lot of things change.

And one of them is the number of chips that communicate together.

where people usually think about this in the context of training.

You know, you have these giant training clusters, you have Colossus with over 100,000 GPUs that are all networked together.

And the inference side, today people usually think about it as an eight-chip cluster or maybe just, you know, NVL 72, the scale-up domain.

this is going to become thousands of chips and tens of thousands of chips.

And the way to get the most performance there, the time between sending data from one chip to another, that primitive matters way more than is getting credit right now.

So when we think about optimizing memory bandwidth for the system, you have to think about how fast these chips can communicate together, because if they can only communicate really quickly with themselves and very slowly with other chips, you're not going to actually be able to serve giant models at 10,000, 20,000 tokens per second.

So we need multiple orders of magnitude of infrastructure build out throughout the entire stack, from the wafer to the watt, transistor to the token, to actually bring this stuff to the world.

Yeah.

And also just certain products aren't usable if they're slow.

So if you want to serve coding models and you want people to actually use them, like there's a certain number of tokens per second you need to hit.

So the question is, while maintaining that per token speed, how many users can I serve at the same time?

And you can basically decide I'm going to shut off a bunch of the world from using this stuff or everyone's going to get a worse experience.

So fundamentally, you need to find ways to push out the curve.

And that's why there's such a pressure for new hardware.

It starts back actually in high school for me.

I've been very unlucky and lucky at different points in life.

This was one of the tougher times.

At the end of my sophomore year of high school, I got injured at a martial arts tournament the next day.

Showing 81–100 of 529 · page 5 of 27