Lance Martin
speaker
624 appearances
1 recordings
1 series
first heard Sep 2025
last heard Sep 2025
Lance Martin’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
So there's a great talk from Hyungwon Chung, previously OpenAI, now at MSL, on the bitter lesson in his approach to AI research.
The take is, compute 10Xs every five years for the same cost, of course.
We all know that.
The history of machine learning has shown, yeah, exactly this slide, exactly.
History of machine learning has shown that actually capturing this scaling is the most important thing.
In particular, algorithms that are more general with fewer inductive biases and more data and compute tend to beat algorithms with more, for example, hand-tuned features, inductive biases built in, which is to say, just letting a machine learn how to think itself
with more compute and data, rather than trying to teach a machine how we think tends to be better.
So that's kind of the bitter lesson piece simply stated.
So his argument is this subtle point that at any point in time, when you're, for example, doing research, you typically need to add some amount of structure to get the performance you want at a given level of compute.
But over time, that structure can bottleneck your further progress.
And that's kind of what he's showing here is that in the low compute regime, kind of on the left of that x-axis, adding more structure, for example, more modeling assumptions, more inductive biases, is better than less.
But as compute grows, less structure, and this is exactly the better lesson point, less structure, more general, tends to win out.
So his argument was we should add structure at a given point in time
in order to get something to work with the level of compute that we have today, but remember to move it later.
And a lot of his argument was, like, people often forget to remove that structure later.
And I think my link here is that I think this applies to AI engineering, too.
And if you kind of scroll down, I have the same chart showing my little... Exactly.
This is my little example of building deep research over the course of a year.
So I started...
With a highly structured research workflow, didn't use tool calling.
Showing 461–480 of 624 · page 24 of 32
← Previous
Next →