Menu
Sign In Search Podcasts Libraries Charts People & Topics Add Podcast API Blog Pricing

Jeremiah

๐Ÿ‘ค Speaker
1265 total appearances

Appearances Over Time

Podcast Appearances

Astral Codex Ten Podcast
SOTA On Bay Area House Party

Did you hear about the guy who made a working language model in Minecraft using redstone circuits?

Astral Codex Ten Podcast
SOTA On Bay Area House Party

Pretty amazing, isn't it?

Astral Codex Ten Podcast
SOTA On Bay Area House Party

His version is barely GPT-2 level, but there's no reason we can't scale that up.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

Once we create full-sized data centers in Minecraft, everyone will want to do their training runs there.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

What do you mean, why?

Astral Codex Ten Podcast
SOTA On Bay Area House Party

Real world data centers cost billions of dollars, raise electricity prices, waste, she briefly scans the room to confirm Andy Masley isn't around, then continues, water, and they're getting increasingly politically unpopular and hard to build.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

We can short-circuit all of that by putting the data centers in Minecraft instead.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

But you have to have the Minecraft world being simulated by real computers, right?

Astral Codex Ten Podcast
SOTA On Bay Area House Party

So don't you still need the data center in order to play the Minecraft?

Astral Codex Ten Podcast
SOTA On Bay Area House Party

Oh, I'm sure you need some computer, but it's a question of leverage.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

One high-end gaming computer playing Minecraft can include a whole world with continents, mountain ranges, forests, and oceans.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

You can fit thousands of data centers in that world.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

So with even one real-world computer, you've saved billions on chips and construction costs.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

You take a moment to consider how best to explain this.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

So, um, every computation has to be done somewhere, right?

Astral Codex Ten Podcast
SOTA On Bay Area House Party

So you can, in theory, build a working data center on Minecraft, but it will take billions of blocks.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

Oh, no problem.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

We've got clawed code working on it.

Astral Codex Ten Podcast
SOTA On Bay Area House Party

No, I'm saying it would take billions of blocks, and simulating the training circuit and all those billions of blocks in perfect detail would take just as many real-world computations as running the training in the real world.