Yoshua Bengio
speaker
2,216 appearances
5 recordings
5 series
first heard Oct 2018
last heard 7 May
Yoshua Bengio’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 3 in all, peaking in May 2026 with 1.
Appearances
but has less guarantees is to take an existing pre-trained model, maybe starting from a base model rather than the one with RL, and then fine tune it using the scientist AI objective and data representation.
And that would give a much more competent model because it's bigger.
Fine-tuning is much cheaper, as you know, than training from scratch, but we lose the mathematical guarantee.
I think it's probably going to be fine anyway.
Of course, it depends how much fine-tuning you're willing to do.
And what's interesting here in these kinds of experiments is that we should be able to see the trade-off.
Like if you measure, say, on deception benchmarks, what happens as you continue training with more and more fine-tuning, we should see a curve that it gets better, right?
And that's what we're hoping to see.
And then you also want to show that capability doesn't go down, which by the way is tricky because unfortunately what we found already in our experiments is that most of the, at least the open weight models, they cheat on the benchmarks.
So what do I mean by this?
As soon as you do a little bit of fine tuning on anything, their performance on the benchmarks goes down.
They probably have overfitted the benchmarks.
So we need to find a way around that, but I'm confident this can be done.
Yeah, so we will have these two kinds of evidence, I hope, and that may be sufficient to convince people to put not just hundreds of millions, but the billions that would be necessary to do a full-scale model from scratch.
So in terms of capability, I would expect it to be better because of better reasoning.
And one aspect I didn't mention yet is there's good scientific evidence that
when a model exploits the causal structure of the world, it can generalize better out of distribution.
So this is something I've worked on and many people in the machine learning community have been working on.
And it has to do with a very interesting concept that the world changes
But somehow there are things that don't change, like the underlying causal mechanisms, like how the world works, like the laws of physics.
Showing 561–580 of 2,216 · page 29 of 111
← Previous
Next →